In today’s work environment there is a need to show changes, potential risks, improved performance, etc. in all areas of the company’s operations. Security professionals need to be prepared to answer the basic question, “why should the CIO or CEO care about security?” CSO Online has a great quote from the post, “From the CIO: Why You Didn’t Get the CISO Job” that challenges us to consider our views when it comes to security. The post states, “laser focus on your speciality is great in middle management. It’s what we want. One of the really hard things about jumping from management to executive is a focus on the whole of the business. It’s a rare person who manages it quickly or easily.” That is basically the problem with metrics. It is a battle between generalization to the point of uselessness and details to the point of not being understandable or collectible. At the end of the day, something needs to be done because the security industry is currently leaving upper management in the position of not understanding what is going on within their business. That is a risk that not acceptable.

Andrew’s article discusses what kind of security metrics should be used. Additional sources of information on security metrics can be found in a previous post entitled “Security Metrics.” The post provides links to wonderful sources on security metric information. You might also want to take a look at the CIS Consensus Security Metrics v1.0.0 guide, NIST Special Publication (SP) 800-55 Rev 1 “Security Metrics Guide for Information Technology Systems”, NIST IR-7564 “Directions in Security Metrics Research”, “Twenty Most Important Controls and Metrics for Effective Cyber Defense and Continuous FISMA Compliance,” and “Metrics, measures & Myths.” Once you have start gathering metrics, you will want to present them in an easy to understand format. This is where Google Visualization can help.

Today’s post walks through an example using the data from the National Institute of Standards and Technology (NIST) National Vulnerability Database (NVD) Common Vulnerabilities and Exposures (CVE) database. The purpose is to provide a working example from which you can learn and apply to the various metrics gathered at your organization.

Data Source

A previous post, “Standardization and Interoperability in Security,” discussed how the Security Content Automation Protocol (SCAP) is an attempt to help defenders by providing a collection of XML schemas/standards that allow technical security information to be exchanged between tools. SCAP components consists of:

• Common Configuration Enumeration (CCE): provide unique identifiers to system configuration issues in order to facilitate fast and accurate correlation of configuration data across multiple information sources and tools.
• Common Platform Enumeration (CPE): a structured naming scheme for information technology systems, platforms, and packages.
• Common Vulnerability Enumeration (CVE): a dictionary of publicly known information security vulnerabilities and exposures.
• Common Vulnerability Scoring System (CVSS): a vulnerability scoring system designed to provide an open and standardized method of rating IT vulnerabilities. NIST has even provided a calculator for creating CVSS vulnerability severity scores.
• eXtensible Checklist Configuration Description Format (XCCDF): a specification language for writing security checklists, benchmarks, and related kinds of documents. NIST has released the NIST Interagency Report 7275 Revision 3 “Specification for Extensible Configuration Checklist Description Format (XCCDF) Version 1.1.4.”
• Open Vulnerability Assessment Language (OVAL): an information security community standard to promote open and publicly available security content, and to standardize the transfer of this information across security tools and services.

We are going to make use of the data from NVD/CVE XML feed with the Common Vulnerability Scoring System (CVSS) mappings (version 2.0). NIST documentation states:

CVSS provides an open framework for communicating the characteristics and impacts of IT vulnerabilities. Its quantitative model ensures repeatable accurate measurement while enabling users to see the underlying vulnerability characteristics that were used to generate the scores. Thus, CVSS is well suited as a standard measurement system for industries, organizations, and governments that need accurate and consistent vulnerability impact scores. Two common uses of CVSS are prioritization of vulnerability remediation activities and in calculating the severity of vulnerabilities discovered on one’s systems.

NVD provides CVSS ‘base scores‘ representing the innate characteristics of each vulnerability. ‘Temporal scores,’ which change over time due to events external to the vulnerability, are not provided though NVD does provide a CVSS score calculator. This allows an organization to add temporal data and even factor in ‘environmental scores‘ customized to reflect the impact of the vulnerability on the organization. Please refer to the CVSS standards guide and the OWASP Risk Rating Methodology concerning factors involved in estimating the severity of risks to your business.

NVD CVE XML Schema

For our example, we will be using the data feeds nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml. Examining the CVE XML 2.0 Schema, we are particularly interested in certain vulnerability and CVSS scoring information. For example, for CVE-2010-1228, we will parse and pull the following kind of information:

<entry id="CVE-2010-1228">
  <vuln:cve-id>CVE-2010-1228</vuln:cve-id>
  <vuln:published-datetime>2010-04-01T18:30:00.453-04:00
  </vuln:published-datetime>
  <vuln:last-modified-datetime>2010-04-05T00:00:00.000-04:00
  </vuln:last-modified-datetime>
  <vuln:cvss>
    <cvss:base_metrics>
      <cvss:score>10.0</cvss:score>
      <cvss:access-vector>NETWORK</cvss:access-vector>
      <cvss:access-complexity>LOW</cvss:access-complexity>
      <cvss:authentication>NONE</cvss:authentication>
      <cvss:confidentiality-impact>COMPLETE</cvss:confidentiality-impact>
      <cvss:integrity-impact>COMPLETE</cvss:integrity-impact>
      <cvss:availability-impact>COMPLETE</cvss:availability-impact>
      <cvss:source>http://nvd.nist.gov</cvss:source>
    </cvss:base_metrics>
  </vuln:cvss>
</entry>

Using Perl to Retrieve the CVE File

Initially we will read the nvdcve-2.0-2010.xml and nvdcve-2.0-2009.xml files. If we start retrieving the file regularly, we would want to change this to nvdcve-2.0-recent.xml. Of course, previous years can also be read in to provide a longer perspective on vulnerabilities. A simple example of a Perl subroutine to read the NVD CVE file and save it locally would be:

sub readpage {
   my($url,$nvd_file) = @_;
   my($proxy) = "http://your-proxy-server:proxy-port";
   my $ua = new LWP::UserAgent;
   $ua->proxy(http  => $proxy);
   $ua->proxy(ftp => $proxy);
   $ua->proxy(https => $proxy);
   # Go out and retrieve page
   my $req = new HTTP::Request('GET', $url);
   my $res = $ua->request($req);
   my $pjstatus = 1;
   # Check if the requested webpage is there and return results
   if ($res->is_success) { # Request successful
       open(OUTFILE,">$nvd_file") || ($pjstatus = 0);
       if ($pjstatus) {
          print OUTFILE $res->content;
       }
       close(OUTFILE);
   }
   else {
      $pjstatus = 0;
   }
   return($pjstatus);
}

Please substitute “http://your-proxy-server:proxy-port” with your site’s proxy server and port, if applicable.

Creating a MYSQL Table to Hold the Data

There is a great deal of information in the NVD CVE file. You will need to determine what information your organization will be interested in storing and graphing. For better or worse, folks have come to expect vulnerabilities to have a “Low,” “Medium,” or “High” score. NIST has stated concerning the NVD Vulnerability Severity Ratings:

NVD provides severity rankings of “Low,” “Medium,” and “High” in addition to the numeric CVSS scores but these qualitative rankings are simply mapped from the numeric CVSS scores:
1. Vulnerabilities are labeled “Low” severity if they have a CVSS base score of 0.0-3.9.
2. Vulnerabilities will be labeled “Medium” severity if they have a base CVSS score of 4.0-6.9.
3. Vulnerabilities will be labeled “High” severity if they have a CVSS base score of 7.0-10.0.

While preferring quantitative over qualitative values, for this example I would like to create a stacked column chart. We will add a severity column which is based on the CVSS score. An example table follows:

CREATE DATABASE vulnerabilities;
USE vulnerabilities;
DROP TABLE IF EXISTS `nvdcve`;
CREATE TABLE `nvdcve` (
  `cve_id` varchar(13) NOT NULL,
  `published` datetime default NULL,
  `modified` datetime default NULL,
  `score` DECIMAL(5,2) default '0.0',
  `severity` varchar(6) default 'LOW',
  `vector` varchar(25) default NULL,
  `complexity` varchar(25) default NULL,
  `authentication` varchar(25) default NULL,
  `confidentiality` varchar(25) default 'NONE',
  `integrity` varchar(25) default 'NONE',
  `availability` varchar(25) default 'NONE',
  `summary` varchar(512) default NULL,
  PRIMARY KEY  (`cve_id`),
  INDEX (score),
  INDEX (vector)
)

Using Perl Populating the Database

Populating the database table is simply a matter of reading the file and adding the entries to the table. An example Perl subroutine follows:

sub readxml {
   my($nvd_file, $dbh) = @_;
   my $parser = XML::LibXML-> new();
   my $doc    = $parser-> parse_file($nvd_file);
   my $xc     = XML::LibXML::XPathContext-> new( $doc->documentElement() );
   $xc-> registerNs(
      def  => 'http://scap.nist.gov/schema/feed/vulnerability/2.0' );
   $xc-> registerNs(
     vuln => 'http://scap.nist.gov/schema/vulnerability/0.4' );
   $xc-> registerNs( cvss => 'http://scap.nist.gov/schema/cvss-v2/0.2' );
   for my $entry ($xc-> findnodes("/def:nvd/def:entry")) {
      my $cve = $xc-> find('vuln:cve-id',$entry);
      my $published = $xc-> find('vuln:published-datetime', $entry);
      my $modified = $xc-> find('vuln:last-modified-datetime', $entry);
      my $summary = $xc-> find('vuln:summary', $entry);
      my $skip = 0;
      my ($metrics) = $xc-> findnodes('vuln:cvss/cvss:base_metrics', $entry) or ($skip = 1);
      if (! $skip) {
         my $score = $xc-> find('cvss:score', $metrics);
         my $vector = $xc-> find('cvss:access-vector', $metrics);
         my $complexity = $xc-> find('cvss:access-complexity', $metrics);
         my $authentication = $xc-> find('cvss:authentication', $metrics);
         my $confidentiality =
            $xc-> find('cvss:confidentiality-impact', $metrics);
         my $integrity = $xc-> find('cvss:integrity-impact', $metrics);
         my $availability = $xc-> find('cvss:availability-impact', $metrics);
         my $severity = "LOW";
         if (int($score) >= 7) {
            $severity = "HIGH";
         }
         elsif (int($score) >= 4) {
            $severity = "MEDIUM";
         }
         my $sql = qq{ SELECT count(*) FROM nvdcve WHERE cve_id=? };
         my $sth = $dbh->prepare( $sql );
         my $rc = $sth->execute($cve);
         if ( $rc) {
            my($exist) = $sth->fetchrow_array();
            if (! $exist) {
                $sql = qq{ INSERT INTO nvdcve SET cve_id=?,
published=?, modified=?, score=?, severity=?, vector=?, complexity=?,
authentication=?, confidentiality=?, integrity=?,availability=?, summary=? };
               $sth = $dbh->prepare( $sql );
               $rc = $sth->execute($cve,$published,$modified,$score,
$severity,$vector,$complexity,$authentication,
$confidentiality,$integrity,$availability,$summary);
            }
         }
      }
   }
}

The Perl Program to Pull It All Together

The above subroutines use the Perl modules LWP::UserAgent, XML::LibXML, XML::LibXML::XPathContext, and DBI. A sample Perl program that calls the above subroutines to pull down the NVD CVE data and load it into a MySQL table would be:

#!/usr/local/bin/perl -w
use LWP::UserAgent;
use XML::LibXML;
use XML::LibXML::XPathContext;
use DBI;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use nvdsubs qw($db_host $db $mysql_user $mysql_passwd $mysql.sock
readpage readxml );
# Main
my $datadir = "/home/johngerber/projects/nvd/data";
my @timeData = localtime(time);
my $year = 1900 + $timeData[5];
my $prev_year = 1900 + $timeData[5] - 1;
my $url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $year . ".xml";
my $prev_url = "http://static.nvd.nist.gov/feeds/xml/cve/nvdcve-2.0-" .
    $prev_year . ".xml";
my $nvd_file = $datadir  . "/nvdcve-". $year . ".xml";
my $prev_nvd_file = $datadir  . "/nvdcve-". $prev_year . ".xml";
$db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:mysql_socket=$mysql.sock;$db:$db_host",
$mysql_user, $mysql_passwd) || die "ERROR: Connecting: $DBI::errstr\n";
my ($pjstatus) = &readpage($prev_url,$prev_nvd_file);
if ($pjstatus) {
   &readxml($prev_nvd_file,$dbh);
}
$pjstatus = &readpage($url,$nvd_file);
if ($pjstatus) {
   &readxml($nvd_file,$dbh);
}
exit;

The nvdsubs.pm file will not be included in this post. The subroutines are defined and the only pieces missing are the MySQL database username and password. You don’t need mine. Add your own. At this point, we have everything we need to finally use Google Visualization to create a graph.

Google Visualization

We are going to create a Perl program that will read our MySQL nvdcve table and generate the JavaScript that will render our charts on the client’s browser. First, we want to define the JavaScript we want to produce. Just to alleviate some concerns, with Google Visualization your data is only shared between your server and the client connecting. This is unlike Google Charts where your data is sent to Google where it is made into a chart and the result is sent back. Google states concerning the logging of chart data (via Google Charts), “The chart data included in the HTTP request is saved in temporary logs for no longer than two weeks for internal testing and debugging purposes.” Every example in the Google Visualization Gallery will state the data policy. For Google Charts, stated at the bottom of the page for each gadget description the data policy:

While Google Visualization gadgets will have the following stated data policy:

Loading Google Libraries

The first thing the JavaScript needs to do is load the required libraries. This is accomplished with the lines:

<script type="text/javascript" src="http://www.google.com/jsapi"></script>

Area Chart and Table

In this example we are going to create an column chart. In a later section, “Other Charting Options” (see below) we define different Google Visualization charting options.

JavaScript code for a sample column chart would be:

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
           [new Date(2009, 0, 30),92,97,3],
           [new Date(2009, 1, 27),168,142,25],
           [new Date(2009, 2, 31),141,165,9],
           [new Date(2009, 3, 30),132,203,12],
           [new Date(2009, 4, 29),158,153,8],
           [new Date(2009, 5, 30),200,199,22],
           [new Date(2009, 6, 31),190,195,11],
           [new Date(2009, 7, 31),127,139,14],
           [new Date(2009, 8, 30),233,208,14],
           [new Date(2009, 9, 30),163,167,18],
           [new Date(2009, 10, 30),129,172,8],
           [new Date(2009, 11, 31),200,211,19],
           [new Date(2010, 0, 29),157,139,14],
           [new Date(2010, 1, 26),137,143,12],
           [new Date(2010, 2, 31),252,242,18],
           [new Date(2010, 3, 13),92,118,17]
        ]);
        var chart = new google.visualization.ColumnChart(document.getElementById('s4graph'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0,
          allowHtml: true, colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'}, {color:'#FFE87C', darker:'#C8B560'}]});
      }
    </script>

The resulting image would be the following column chart:

Rendering the Table

When providing qualitative results, I like to back them up with more accurate numeric values. Let us include a table with links to the CVSS scores for each vulnerability.

    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
           [{v:new Date(2009, 0, 30),
              f:'<a href="/nvd/cvealerts.php?date=2009-01">2009-01-30</a>'}, 92,97,3],
           [{v:new Date(2009, 1, 27),
              f:'<a href="/nvd/cvealerts.php?date=2009-02">2009-02-27</a>'}, 168,142,25],
           [{v:new Date(2009, 2, 31),
              f:'<a href="/nvd/cvealerts.php?date=2009-03">2009-03-31</a>'}, 141,165,9],
           [{v:new Date(2009, 3, 30),
              f:'<a href="/nvd/cvealerts.php?date=2009-04">2009-04-30</a>'}, 132,203,12],
           [{v:new Date(2009, 4, 29),
              f:'<a href="/nvd/cvealerts.php?date=2009-05">2009-05-29</a>'}, 158,153,8],
           [{v:new Date(2009, 5, 30),
              f:'<a href="/nvd/cvealerts.php?date=2009-06">2009-06-30</a>'}, 200,199,22],
           [{v:new Date(2009, 6, 31),
              f:'<a href="/nvd/cvealerts.php?date=2009-07">2009-07-31</a>'}, 190,195,11],
           [{v:new Date(2009, 7, 31),
              f:'<a href="/nvd/cvealerts.php?date=2009-08">2009-08-31</a>'}, 127,139,14],
           [{v:new Date(2009, 8, 30),
              f:'<a href="/nvd/cvealerts.php?date=2009-09">2009-09-30</a>'}, 233,208,14],
           [{v:new Date(2009, 9, 30),
              f:'<a href="/nvd/cvealerts.php?date=2009-10">2009-10-30</a>'}, 163,167,18],
           [{v:new Date(2009, 10, 30),
              f:'<a href="/nvd/cvealerts.php?date=2009-11">2009-11-30</a>'}, 129,172,8],
           [{v:new Date(2009, 11, 31),
              f:'<a href="/nvd/cvealerts.php?date=2009-12">2009-12-31</a>'}, 200,211,19],
           [{v:new Date(2010, 0, 29),
              f:'<a href="/nvd/cvealerts.php?date=2010-01">2010-01-29</a>'}, 157,139,14],
           [{v:new Date(2010, 1, 26),
              f:'<a href="/nvd/cvealerts.php?date=2010-02">2010-02-26</a>'}, 137,143,12],
           [{v:new Date(2010, 2, 31),
              f:'<a href="/nvd/cvealerts.php?date=2010-03">2010-03-31</a>'}, 252,242,18],
           [{v:new Date(2010, 3, 13),
              f:'<a href="/nvd/cvealerts.php?date=2010-04">2010-04-13</a>'}, 92,118,17],
        ]);
        var table = new google.visualization.Table(document.getElementById('s4graph_tab'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
      }
    </script>

The JavaScript code assumes there is a PHP program called cvealerts.php under the /nvd directory on your web server. Adjust to your environment. A sample PHP program that could be used for cvealerts.php is provided below. The resulting table chart would look like:

Handling Events: Interactions Between Graphs

We now have two different types of graphs representing the same data. We want to add interaction between the graphs so the viewer can see the relationship. With tables rows are selected when the user clicks, which correspond to the whole column of the stacked column chart. It is not a perfect fit, but it does demonstrate nicely use of adding interactions.

        // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
        // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });

Providing Detailed Information

When the table chart link is clicked, we would like to provide some detailed information about the vulnerability. For this example, we will do this with a simple PHP program placed in the /nvd directory on the web server. The program is called cvealerts.php.

<?
session_start();
function db_connect($table) {
   $result = mysql_pconnect("<dbhost>:<dbport>", "<username>", "<password>");
   if (!$result) return false;
   if (!mysql_select_db($table)) return false;
   return $result;
}
function do_html_header($title,$checkuser,$logpage) {
?>
  <html> <head> <title><?=$title?></title></head>
  <body bgcolor="#FFFFFF">
<?
}
function do_html_footer() {
?>
<table>
<tr><td ALIGN=CENTER NOWRAP WIDTH="590"></font>
<font face="Verdana, Arial, Helvetica" size=-2>Notice to Users: Use
of this system constitutes consent to security monitoring and testing.
<br>All activity is logged with your host name and IP address.</font>
</td></tr>
</table>
</body>
 </html>
<?
}
// Main
$dates= array();
$stringlist = "";
if (isset($_GET['date'])) {
    $passdates = explode(",",$_GET['date']);
    for ($index=0; $index<count($passdates); $index++) {
       array_push($dates, $passdates[$index]);
       $stringlist .= $passdates[$index] . " ";
    }
}
else {
  print("Confusion over how you arrived at this page.<P>\n");
  exit;
}
$stringlist = preg_replace("/ $/", "",$stringlist);
do_html_header("Review NVD CVE Announcements for Month Ending $stringlist",1,1);
$nvd_host = "http://web.nvd.nist.gov/view/vuln/detail?vulnId=";
$conn = db_connect("vulnerabilities");
if (!$conn)
   logit("Could not connect to database vulnerabilities - please try later.\n",1);
for ($index=0; $index<count($dates); $index++) {
   $rule = $dates[$index];
   $sql = "SELECT cve_id,score,published,vector,severity,complexity,left(summary,50)
    FROM vulnerabilities.nvdcve
      WHERE date_format(published,'%Y-%m')='$rule'
       ORDER BY (score+0)";
   $result = mysql_query($sql,$conn);
   if (!$result)
       logit("Problem with $sql\n",1);
   print("<table border=1><tr><td><table border=0><tr><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Bulletin</font></th><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Impact</font></th><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Date</font></th><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Vector</font></th><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Severity</font></th><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Complexity</font></th><th bgcolor=\"#727D96\">
<font color=\"#ffffff\" face=\"arial,helvetica,sanserif\">Short Summary</font></th></tr>\n");
   for ($count = 1; list($cve_id, $score, $date, $vector, $severity,$complexity,$shortsum) =
     mysql_fetch_array ($result, MYSQL_NUM); ++$count) {
?>
      <tr><td CLASS="plfieldhdrleft" WIDTH="20%" BGCOLOR='#F0F5FF'>
      <?  print("<a href=\"$nvd_host$cve_id\">$cve_id</a>"); ?>
      </td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
      <?  print($score); ?>
      </td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
      <?  print($date); ?>
      </td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
      <?  print($vector); ?>
      </td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
      <?  print($severity); ?>
      </td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F9FCFF'>
      <?  print($complexity); ?>
      </td>
      <td CLASS="plfieldhdrleft" BGCOLOR='#F0F5FF'>
      <?  print($shortsum); ?>
      </td>
      </tr>
<?
   }
}
print("</table></td></tr></table>");
do_html_footer();

The PHP program would generate a HTML table displaying the NVD CVE alerts for that month. The table would look like:

When the CVE link is clicked on, the user is taken to the NIST NVD site where additional information is available.

Using Perl to Create the JavaScript

The Perl code is rather simple now that we have the MySQL tables defined and the JavaScript we want to generate. Much of the code consists of the JavaScript listed above.

#!/usr/local/bin/perl -w
use DBI;
use Time::Local;
use POSIX qw(strftime);
use LWP::UserAgent;
BEGIN{push @INC, "/home/jgerber/projects/nvd/perl"}
use ornl_feds qw($db_host $db $mysql_user $mysql_passwd );
sub slide_nvd_alerts {
  my($min_date,$graph_name,$web_link,$dbh) = @_;
  my $slide = "";
  my $slide_head = qq!
    <script type='text/javascript'>
      google.load('visualization', '1', {packages:['columnchart,table']});
      google.setOnLoadCallback(drawChart);
      function drawChart() {
        var data = new google.visualization.DataTable();
        data.addColumn('date', 'Date');
        data.addColumn('number', 'High');
        data.addColumn('number', 'Medium');
        data.addColumn('number', 'Low');
        data.addRows([
!;
   my $slide_head_table = qq!
        var data2 = new google.visualization.DataTable();
        data2.addColumn('date', 'Date');
        data2.addColumn('number', 'High');
        data2.addColumn('number', 'Medium');
        data2.addColumn('number', 'Low');
        data2.addRows([
!;
   my $table_div = $graph_name . "_tab";
   my $slide_tail = qq!
        var chart = new google.visualization.ColumnChart(document.getElementById('$graph_name'));
        chart.draw(data, {displayAnnotations:true, is3D: true, isStacked: true, min: 0, allowHtml: true,
 colors:[{color:'#E41B17', darker:'#C11B17'}, {color:'#FFA500', darker:'#E56717'},
{color:'#FFE87C', darker:'#C8B560'}]});
        var table = new google.visualization.Table(document.getElementById('$table_div'));
        table.draw(data2, {showRowNumber: true, sortAscending: false, sortColumn: 0, allowHtml: true});
            // Set a 'select' event listener for the table.
        // When the table is selected,
        // we set the selection on the line graph.
        google.visualization.events.addListener(table, 'select', function() {
          chart.setSelection([{row: table.getSelection()[0].row, column: 1}]);
         });
      // Set a 'select' event listener for the graph.
        // When the graph is selected,
        // we set the selection on the table.
        google.visualization.events.addListener(chart, 'select', function() {
           table.setSelection([{row: chart.getSelection()[0].row}]);
        });
      }
    </script>
!;
   if ($min_date eq "") {
      my $sql2 = qq{ SELECT min(published) FROM vulnerabilities.nvdcve };
      my $sth2 = $dbh->prepare( $sql2 );
      my $rc2 = $sth2->execute();
      if ($rc2) {
         $min_date = $sth2->fetchrow_array();
      }
   }
   my $table_data = "";
   my $graph_data = "";
   my $sql2 = qq{ select date_format(published,'%Y-%m'),severity,count(severity)
      FROM vulnerabilities.nvdcve where published >= ? group by date_format(published,'%Y-%m'),severity };
   my $sth2 = $dbh->prepare( $sql2 );
   my $rc2 = $sth2->execute($min_date);
   if ($rc2) {
      my ($change,$virgin,$ht,$mt,$lt,$mmax_date) = ("",1,0,0,0,"");
      while (my($snapshot_date, $severity, $pcount) = $sth2->fetchrow_array()) {
         my $sql3 = qq{ SELECT max(published) FROM vulnerabilities.nvdcve where
date_format(published,'%Y-%m')=? };
         my $sth3 = $dbh->prepare( $sql3 );
         my $rc3 = $sth3->execute($snapshot_date);
         $max_date =  $sth3->fetchrow_array();
         $max_date =~ s/ \S+$//;
         if ($change ne $snapshot_date) {
            if (! $virgin) {
                my($year,$month,$day) = split("-",$mmax_date);
                my $mmonth = $month;
                $month--;
                $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt],
!;
                $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'<a href="$web_link/cvealerts.php?date=$year-$mmonth">$mmax_date</a>'}, $ht,$mt,$lt],
!;
                ($ht,$mt,$lt) = (0,0,0);
             }
             $change = $snapshot_date;
          }
          if ($severity eq "HIGH") { $ht = $pcount; }
          elsif ($severity eq "MEDIUM") { $mt = $pcount; }
          elsif ($severity eq "LOW") { $lt = $pcount; }
          if ($mmax_date eq "") { $mmax_date = $max_date; }
          if ($mmax_date lt $max_date) { $mmax_date = $max_date; }
          $virgin = 0;
      }
      my($year,$month,$day) = split("-",$mmax_date);
      my $mmonth = $month;
      $month--;
      $graph_data .= qq!           [new Date($year, $month, $day),$ht,$mt,$lt]
!;
     $table_data .= qq!           [{v:new Date($year, $month, $day),
              f:'<a href="$web_link/cvealerts.php?date=$year-$mmonth">$mmax_date</a>'}, $ht,$mt,$lt],
!;
   }
   $table_data .= "        ]);\n";
   $graph_data .= "        ]);\n";
   $slide = $slide_head .  $graph_data . $slide_head_table . $table_data . $slide_tail;
   return($slide);
}
sub slide_body {
  my($graph_name,$title,$style) = @_;
  my $table_name = $graph_name . "_tab";
  my $table_text = "div id=\"$table_name\"";
  if ($style ne "") {
     $table_text .= " style=\'$style\'";
  }
  my $slide2 = "<h3>$title</h3>\n";
  my $itext = "div id=\"$graph_name\"";
  if ($style ne "") {
     $itext .= " style=\'$style\'";
  }
  $slide2 .= qq{
    <table><tr>
    <td valign="top"><$itext></div></td>
    <td valign="top"><$table_text></div></td>
    <td valign="top">   </td>
    <td valign="top"><div id="labels"></div></td>
    </tr></table>
  };
  return($slide2);
}
# Main
my $web_link = "/nvd";
my $results_dir = "/data/html" . $web_link;
my $result_file = $results_dir . "/nvdcve_stats.html";
my $debug = 1;
my $db = "vulnerabilities";
local($dbh) = DBI->connect("DBI:mysql:$db:$db_host", $mysql_user, $mysql_passwd) ||
   die "ERROR: Connecting: $DBI::errstr\n";
$slides_data .= &slide_body("s4graph","NVD CVE Alerts","width:700px; height:400px;");
$slides_head .= &slide_nvd_alerts("","s4graph",$web_link,$dbh);
open(OUTFILE,">$result_file");
print OUTFILE "<HTML>\n<HEAD><TITLE>NVD CVE Statistics</TITLE>\n";
print OUTFILE "<script type=\"text/javascript\" src=\"http://www.google.com/jsapi\"></script>\n";
print OUTFILE $slides_head;
print OUTFILE "</HEAD>\n<BODY>\n";
print OUTFILE $slides_data;
print OUTFILE "</BODY>\n";
close(OUTFILE);
exit;

Other Charting Options

Google, Google users, and other companies have shared some JavaScript visualizations built on the Google Visualization API to help you get started. Below are some example:

	Annotated Time Line (GWT Integrated) An animated time series chart. By: Google
	Area Chart (GWT Integrated) Interactive area chart. By: Google
	Bar Chart (GWT Integrated) Interactive bar chart. By: Google
	Bars of Stuff Fun bar charts using images of trains, chocolate, worms, and more. By: The visapi project
	Bio Heat Map Heatmaps are a useful way to visualize matricies of data. Scientists often use green-black-red heatmaps to visualize gene expression data from microarrays. This visualization supports both three color heatmaps (ex: green to black to red) and two color heatmaps (ex: white to yellow). By: Institute for Systems Biology
	Column Chart (GWT Integrated) Interactive column chart. By: Google
	Drastic Treemap A dynamic treemap in Flash. By: DrasticData
	Dygraphs The dygraphs JavaScript library produces interactive, zoomable charts of time series. By: Dan Vanderkam
	Filters A Visualization that acts as a control over other visualizations. It is rendered within the browser using HTML. This visualization offers the ability to select some criteria to filter the DataTable used by the controlled visualizations. By: Institute for Systems Biology
	Gauge (GWT Integrated) Each numeric value is displayed as a gauge. By: Google
	Geo Map (GWT Integrated) A map of a country, continent, or region map, with colors and values assigned to specific regions. Values are displayed as a color scale, and you can specify optional hovertext for regions. By: Google
	Intensity Map (GWT Integrated) An intensity map that highlights regions or countries based on relative values. By: Google
	Line Chart (GWT Integrated) Interactive line chart. By: Google
	Magic-Table The Magic Table is a JavaScript library that allows you to see more in your data by applying some simple visual techniques to transform a table. The table is displayed in the browser by the canvas element. Internet Explorer is not supported. By: Greg Ross
	Map (GWT Integrated) An interactive map that uses the Google Maps API. By: Google
	Motion Chart (GWT Integrated) Motion Chart: A dynamic flash based chart to explore several indicators over time. Required columns: bubble name, time and 2 columns of numeric values. Optional columns: Numeric values or categories. By: Google
	Organizational Chart A GWT Integrated simple organizational chart. By: Google
	Parallel Coordinates Chart Parallel Coordinates is a method of visualizing multivariate data. An n-dimensional space is represented as n parallel lines. Works for browsers based on Gecko or Presto (does not work in IE). This is written in Javascript, no Flash required. By: Sri Harsha Allamraju
	Pie Chart (GWT Integrated) Interactive pie chart. By: Google
	Piles of Money Column chart made of of money bills. By: The visapi project
	Scatter Chart (GWT Integrated) Interactive scatter chart. By: Google
	Table (GWT Integrated) A highly customizable table with sorting, paging and selection capabilities. By: Google
	TermCloud A list of terms, where the size and color of each word is determined by a specified frequency value (typically the number of times it appears in some text). By: The visapi project
	Thematic Mapping API Enables visualization of data in Google Earth or other geobrowsers through the use of the Google Visualization API and KML. By: Thematicmapping.org
	WordCloud Displays all words in text with size and color based on the number of time each word appears. By: The visapi project

Additional Information

Below is the talk that Itai Raz, the lead engineer for the Visualization API product at Google, gave at Google I/O 2009 titled “Using the Visualization API with GWT:”

Additional Possibilities

The work above is meant only to serve as a starting point. There is a great deal more information to expand upon. For example, we began this post pulling some information from the XML schema for CVE-2010-1228. One field we did not pull out from the XML file is:

    <vuln:cwe id="CWE-362" />

The Common Weakness Enumeration (CWE) represents vulnerability types and NIST provides a CWE Cross Section Mapped into by NVD table. In the above example, we see an entry:

Clicking on the link will take us to the MITRE site that provides a great deal more information on CWE entries. It is easy enough to expand on the above program to harvest this information for a richer information database.

Another possibility is to expand the above program to pull additional information on the CVE entry. In additional to the data in the NVD CVE XML file, we could pull information from the NVD site. Using CVE-2010-1228 as an example, we could have the program pull down the page:

http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2010-1228

Notice the line:

CVSS v2 Base Score:10.0 (HIGH) (AV:N/AC:L/Au:N/C:C/I:C/A:C) (legend)

The (AV:N/AC:L/Au:N/C:C/I:C/A:C) provides values that were used in determining the base score. If you follow the link, you will see the values used in the calculations:

• CVSS Base Score: 10
  • Impact Subscore: 10
  • Exploitability Subscore: 10
• CVSS Temporal Score: Undefined
• CVSS Environmental Score: Undefined
• Overall CVSS Score: 10

NVD has made available the equations used in calculating the CVSS base score, temporal score, and environmental score.

Three other pieces of information that might provide interesting groupings are:

• Access Complexity: Low **NOTE: Access Complexity scored Low due to insufficient information
• Authentication: Not required to exploit
• Impact Type: Allows unauthorized disclosure of information; Allows unauthorized modification; Allows disruption of service

What information is of interest and how it is used will be dependent on your organization. There is a great deal of information available and many directions you start examining.

Final Thoughts

I am often reminded of the old phrase, “Trust us, we are from the government.” No one really trusts anyone, especially when it comes to matters they do not understand. Just because you are from the security group at your organization, is that reason enough for the CEO to give you unlimited money and authority to do what you see fit? Of course not. While management might trust you, they may not believe that you are capable of seeing the big picture. That is after all their job.

Another great old saying is that “the devil is in the details.” Those details will likely fall in the security domain. In organization across the planet there is a tug of war between the details and the big picture with multiple groups adding in their opinions and views. You need to make the details understandable to your higher management to effectively argue your view. Finding effective metrics and finding clear representation is essential in today’s business. Google Visualization can be a useful tool in accomplishing this task.

If you are going to be using RT, you need to get the “RT Essentials” book written by Jesse Vincent, Robert Spier, Dave Rolsky, Darren Chamberlain, and Richard Foley. It is a good reference and a quick read. For up-to-date information, see the RT Wiki and the Best Practical Solutions blog site.

Prerequisites

To start, please review the following posts:

1. An Apache Implementation
2. Apache and OpenSSL
3. PHP Implementation
• PHP as a Module
• PHP as a CGI
• PHP Configuration Modifications
4. Introduction to MySQL
5. Setting Up and Securing MySQL: References
6. Implementing a Web Application Firewall with ModSecurity

Install Software

With Apache, MySQL, PHP, OpenSSL, and ModSecurity installed, we are now ready to focus on software packages required by RT.

1. Installing expat.

Different operating systems will vary on whether expat, the XML parser, is installed. Expat is needed to complete the cpan install for XML::RSS. Check your particular operating system.

 root# cd /usr/local/src
 /usr/local/src root# wget http://downloads.sourceforge.net/expat/expat-2.0.1.tar.gz
 /usr/local/src root# tar xzf expat-2.0.1.tar.gz
 /usr/local/src root# cd expat-2.0.1
 /usr/local/src/expat-2.0.1 root# ./configure
 /usr/local/src/expat-2.0.1 root# make
 /usr/local/src/expat-2.0.1 root# make check
 /usr/local/src/expat-2.0.1 root# make install

2. Install FastCGI

For RT, you can install mod_perl or mod_fastcgi. In this posting, we are going to walks through the installation of FastCGI. Information concerning mod_perl will be provided below so the reader can chose what fits best in their environment. FastCGI is much simpler to install and allows the core Apache process to stay small in size. With FastCGI, RT runs as a separate process from Apache allowing RT to be stopped and restarted without affecting the Apache server. In general, FastCGI programs are easier to manage.

The Apache module mod_fastcgi allows a web server to run CGI scripts via a separate, persistent program. PHP comes with FastCGI support compiled in by default, so nothing needs to be done to the PHP installation.

You can have the Apache program call FastCGI, and have it run as the same user as the Apache server or use suexec to have FastCGI switch to a different user. Under some operating systems, suexec may not get compiled and installed when installing Apache. Check if suexec is installed, and if not go back to the Apache source, compile it, and install it. Initially, we are not going to use the suexec program. Instead we will create the group “rt”, add user httpd to group rt, and set permissions that way. You may choose later to use suexec.

 root# ls -la /usr/local/apache/bin/suexec
ls: /usr/local/apache/bin/suexec: No such file or directory
 root# cd /usr/local/src/httpd-2.2.8
 /usr/local/src/httpd-2.2.8 root# make suexec
 /usr/local/src/httpd-2.2.8 root# cp ./support/suexec /usr/local/apache/bin/suexec

Now, we are ready to get mod_fastcgi installed.

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.fastcgi.com/dist/mod_fastcgi-2.4.6.tar.gz
 /usr/local/src root# tar xzf mod_fastcgi-2.4.6.tar.gz
 /usr/local/src root# cd mod_fastcgi-2.4.6
 /usr/local/src/mod_fastcgi-2.4.6 root# cp Makefile.AP2 Makefile
 /usr/local/src/mod_fastcgi-2.4.6 root# make top_dir=/usr/local/apache
 /usr/local/src/mod_fastcgi-2.4.6 root# make top_dir=/usr/local/apache install
 /usr/local/src/mod_fastcgi-2.4.6 root# /usr/local/apache/bin/apachectl stop
 /usr/local/src/mod_fastcgi-2.4.6 root# vi /usr/local/apache/conf/httpd.conf

Add the following lines to the Apache httpd.conf file:

# Load the mod_fastcgi module.
LoadModule fastcgi_module modules/mod_fastcgi.so

Check if installation and configuration is working.

 /usr/local/src/mod_fastcgi-2.4.6 root# /usr/local/apache/bin/apachectl configtest
Syntax OK
 /usr/local/src/mod_fastcgi-2.4.6 root# /usr/local/apache/bin/apachectl start
 /usr/local/src/mod_fastcgi-2.4.6 root# cat /var/www/logs/error_log | grep -i fastcgi
[Fri Aug 01 12:17:22 2008] [notice] FastCGI: process manager initialized (pid 15221)
[Fri Aug 01 12:17:22 2008] [notice] Apache/2.2.8 (Unix) mod_ssl/2.2.8
OpenSSL/0.9.7a mod_fastcgi/2.4.6 configured -- resuming normal operations

For in depth coverage of mod_perl, Stas Bekman and Eric Cholet have written the book, “Practical mod_perl.” They have made the complete book available online in both HTML and PDF format under the Creative Commons Attributes Share-Alike License. Stas Bekman and Jim Brandt have also written the “mod_perl2 User’s Guide Book” where 50% of the book’s proceeds go to The Perl Foundation.

If you are installing under Mac OS X, mod_perl may complain about Perl 5.8.8 being built without threads and you will get a message about building perl with -Duserthreads. If you are determined to use mod_perl, consider dropping back to Apache 1.3.x and using mod_perl 1.x. While Apache 1.3.x is legacy code, and I tend to want to use the code that is being actively developed, there is an argument for using Apache 1.3.x. One major feature of Apache 2.x is threading. On Windows, where most basic libraries are and must be threadsafe, Apache 2 is really the only choice. Earlier Mac OS X releases did not include a completely thread-safe libc, so threading is still not fully supported in Perl. This is why the Perl version that comes with Mac OS X is not compiled to use threads. To use Apache2.x, Perl will need to be configured to use threads. The code is available from the Perl web site.

Rather than getting bogged down in compiling Perl to use thread, we will move ahead and use FastCGI. By the time this post, I will have worked on getting RT installed under Linux, Mac OS X, and FreeBSD. Figuring out what software works best in a multi OS environment can be challenging.

3. Configure RT

Let us start by adding the group RT. Under many operating systems, this would be done with the simple command “groupadd rt.” Things are always more interesting under Mac OS X, where you would have to first look at what group ids (gid), choose an unused gid, and then create the rt group using that gid. Under Mac OS X Leopard, group rt would be created with the commands:

 root# dscl . list /groups PrimaryGroupID | sort -k 2,2 -n
 root# dscl . create /groups/rt gid gid-of-rt
 root# dscl . create /groups/rt passwd '*'
 root# dscl . read /groups/rt
AppleMetaNodeLocation: /Local/Default
Password: *
PrimaryGroupID: gid-of-rt
RecordName: rt
RecordType: dsRecTypeNative:groups

RT’s primary maintenance and documentation site is http://www.bestpractical.com. Documentation can be found at the Best Practical Solutions RT Wiki located at http://wiki.bestpractical.com/. The latest TAR/GZ is located at http://download.bestpractical.com/pub/rt/release/rt.tar.gz. The lack of any version numbers means the version can be updated at any time. The latest version, as of this writing, is 3.8.0.

The following are the steps for downloading and configuring RT:

 root# cd /usr/local/src
 /usr/local/src root# wget http://download.bestpractical.com/pub/rt/release/rt.tar.gz
 /usr/local/src root# tar xzf rt.tar.gz
 /usr/local/src root# cd rt-3.8.0
 /usr/local/src/rt-3.6.5 root# ./configure \
  --with-web-user="httpd" \
  --with-web-group="httpd" \
  --with-rt-user="httpd" \
  --with-rt-group="rt"

4. Install Apache::TEST

Perl module Apache::TEST will not allow you to run the test check as root. You can download the module separately as a non root user and after configuring, compiling, and testing the program, you install it as root.

 root# su - goofy
 ~$ cd src
 ~/src goofy$ wget http://search.cpan.org/CPAN/authors/id/P/PH/PHRED/Apache-Test-1.30.tar.gz
 ~/src goofy$ tar xzf Apache-Test-1.30.tar.gz
 ~/src goofy$ cd Apache-Test-1.30
 ~/src goofy$ perl Makefile.PL
 ~/src goofy$ make
 ~/src goofy$ make test
 ~/src goofy$ sudo su root
 root# make instal

5. Run fixdeps Command and Install Perl Modules

Now you are ready to utilize the fixedeps utility that comes with RT to install required Perl modules. There is also the testdeps utility to test if all dependencies are installed and RT is ready to be installed. You may need to run fixdeps multiple times before testdeps reports that you have all required software packages. The first time through, it can take awhile (depending on your installation). Be aware that some perl modules may need to be installed manually. It various depending on OS and your environment. You will be able to tell which modules need manual installation by the final message provided by the fixdeps program.

 root# cd /usr/local/src/rt-3.8.0
 /usr/local/src/rt-3.8.0 root# make fixdeps
 /usr/local/src/rt-3.8.0 root# make fixdeps
 /usr/local/src/rt-3.8.0 root# make testdeps

6. Install RT

The final installation of RT is the easy part.

 /usr/local/src/rt-3.8.0 root# make install

7. Configure RT_SiteConfig.pm

We now will configure /opt/rt3/etc/RT_SiteConfig.pm. In the next step a database user and a database will be setup. We are only adding those values to the configuration file in this step. I am going to set up a hostname (rt.securitymonks.com) for my current machine. Please do not copy blindly. Change this to your environment. We will create the hostname so it only exists locally by adding an entry into the machines /etc/hosts file. Right now, I am only going to access the Apache server from this machine. In other words, the client and server will be on the same box.

 /usr/local/src/rt-3.8.0 root# vi /etc/hosts

Add the following line, adapting it to your organization:

 /usr/local/src/rt-3.8.0 root# vi /etc/hosts
##
127.0.0.1       localhost
10.1.218.202   rt.securitymonks.com

We are now ready to modify the RT_SiteConfig.pm file.

 /usr/local/src/rt-3.8.0 root# vi /opt/rt3/etc/RT_SiteConfig.pm

At minimum, add the following linesto /opt/rt3/etc/RT_SiteConfig.pm:

Set($rtname, 'BRORT');
Set($Organization, 'securitymonks');

Set($CorrespondAddress , 'john@securitymonks.com');
Set($CommentAddress , 'john@securitymonks.com');

Set($Timezone , 'US/Eastern'); # obviously choose what suits you

# THE DATABASE:

Set($DatabaseType, 'mysql'); # e.g. Pg or mysql

# These are the settings we used above when creating the RT database,
# you MUST set these to what you chose in the section above.

Set($DatabaseUser , 'rtuser');
Set($DatabasePassword , 'secret');
Set($DatabaseName , 'rtdb');

# THE WEBSERVER:

Set($WebPath , '');
Set($WebBaseURL , 'https://rt.securitymonks.com');

# Logging
Set($LogToSyslog, '');
Set($LogToFile, 'debug');
Set($LogDir, '/opt/rt3/var/log');
Set($LogToFileNamed, "rt.log");

8. Initialize the Database

RT needs to create the rtdb database, the rt db users, and initialize some tables. This can be done with the command initialize-database, which should be run only once.

 /usr/local/src/rt-3.8.0 root# make initialize-database
 /usr/local/bin/perl sbin/rt-setup-database --action init --dba root --prompt-for-dba-password
In order to create or update your RT database, this script needs to connect to your mysql
instance on localhost as root.  Please specify that user's database password below. If the
user has no database

password, just press return.

Password:
Working with:
Type:   mysql
Host:   localhost
Name:   rtdb
User:   rtuser
DBA:    root
Now creating a mysql database rtdb for RT.
Done.
Now populating database schema.
Done.
Now inserting database ACLs
Granting access to rtuser@'localhost' on rtdb.
Done.
Now inserting RT core system objects
Done.
Now inserting data
Done inserting data
Done.

Check the MySQL database out.

 /usr/local/src/rt-3.8.0 root# mysql -u rtuser -p
mysql> use rtdb;

9. Modify Apache Configuration File

Edit the /usr/local/apache/conf/httpd.conf file.

 /usr/local/src/rt-3.8.0 root# /usr/local/apache/bin/apachectl stop
 /usr/local/src/rt-3.8.0 root# vi /usr/local/apache/conf/httpd.conf

We are going to have the RT server run under our secure web server. Find the “<virtualhost _default_:443>” line, change it to “<virtualhost 10.1.218.202:443>“. Add the following lines to that section (adjusting to your environment):

   ServerName rt.securitymonks.com
   DocumentRoot /opt/rt3/share/html
   ErrorLog /usr/local/apache/logs/rt.error
   LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
   CustomLog /usr/local/apache/logs/rt.access_log combined
   AddHandler fastcgi-script fcgi
   ScriptAlias / /opt/rt3/bin/mason_handler.fcgi/

Add the user the Apache server runs as (httpd by default), to the RT group. For non Mac OS X, modify group membership by editing the file /etc/group (vi /etc/group). Mac OS X users need to user the dscl command.

 root# dscl . append /groups/rt GroupMembership httpd
 root# dscl . read /groups/rt

Change the group and permission on the log area if you have told RT to log to /opt/rt3/var/log.

 root# chgrp rt /opt/rt3/var/log
 root# chmod g+w /opt/rt3/var/log

Test the configuration of the file, and if everything checks out start up Apache.

 /usr/local/src/rt-3.8.0 root# /usr/local/apache/bin/apachectl configtest
Syntax OK
 /usr/local/src/rt-3.8.0 root# /usr/local/apache/bin/apachectl start

Remember there are now three files to check for problems with RT.

• /opt/rt3/var/log/rt.log
• /usr/local/apache/logs/rt.error
• /usr/local/apache/logs/rt.access_log

There are many configuration operations. The options chosen in this post represents only the minimal to get RT running. Please see the RT Wiki’s FastCGIConfiguration page for additional information.

10. Access RT and Change the Default Password

Now it is time to log in and change the default password. Using the entry we made in our /etc/hosts file, we can now access the site by going to https://rt.securitymonks.com. This URL should be different for your site. You will see a login screen similar to the image on the left.

Log in using the username “root” and password “password“. Once logged in, you will see the screen similar to the image below (click on the image if you need to enlarge):

Over on the left menu bar, select “Configuration.” That will bring you to the “RT Administration” screen:

Select, “Users.” That will bring you to the “Select a user” screen:

Select the user “root,” which will bring you to the “Modify the user root” screen. If you look at the lower left of the screen, there is a “Access Control area.” There is a place to enter “New Password.” Do so. The screen looks like:

Make sure to hit the “Save Changes” button at the bottom of the screen. With a working copy of RT, you are not ready to start adjusting configurations and working with the program. For additional information, Please check out the “RT Essentials” and the RT Wiki and the Best Practical Solutions blog site. Look for future posts to build upon the RT installation and database.

Previously, posts discussed PHP Implementation, PHP as a Module, PHP as a CGI, and PHP over FastCGI. This final post discusses some configuration considerations when implementing PHP. Earlier, we copied the file php.ini-recommended to /usr/local/apache/php/lib/php.ini. There are a few options that need to be configured for a more secure environment. The below tables outline some options whose values should be understood and adjusted to your environment. Links, provided by the PHP Security Consortium, supply additional security considerations. No matter which implementation of PHP you choose, make sure to understand the configuration options so you may run PHP as securely as possible.

In the initial post, “PHP Implementation” three ways to implement PHP were outlined. In the previous post, PHP as a Module, PHP configured to run as an Apache module was examined. Today’s post follows that up with a discussion of configuring PHP as a CGI. There are many options in configuring PHP as a CGI. This post is only meant as an introduction and an example to get you started. Please see previously mentioned source for a more in depth discussion. Configuring PHP as a CGI is not much different from that of a module. There are two methods:

• Treat PHP scripts like other CGI scripts. Execution is carried out by the operating system. The file must be executable to the Apache user. The Apache configuration file must have ExecCGI set. The PHP –enable-discard-path configuration option is used.

On most Unix systems, the first line in the script tells the system how to execute the script. For example: #!/bin/sh, #!/usr/bin/perl, or #!/usr/local/apache/php/bin/php. As an Apache module, PHP scripts do not require this first line so many software packages do not write PHP code in this manner. Therefor, you would have to modify every code currently installed and many PHP code you would install in the future.

• Use the –enable-force-cgi-redirect option and have Apache post-process the CGI scripts.

We are going to use the second method. Please see previous section, “Configuring PHP as a Module” as to the meaning of the additional configuration options.

 /usr/local/src root# cd /usr/local/src/php-5.2.5
 /usr/local/src/php-5.2.5 root# CC=gcc CFLAGS="-O3 -fno-omit-frame-pointer" \
CXX=gcc CXXFLAGS="-O3 -fno-omit-frame-pointer -felide-constructors \
-fno-exceptions -fno-rtti" ./configure --enable-force-cgi-redirect \
--prefix=/usr/local/apache/php --enable-sockets --with-gd --with-mysql=/usr/local/mysql \
--with-zlib-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr
 /usr/local/src/php-5.2.5 root# make
 /usr/local/src/php-5.2.5 root# make install

To avoid the possibility of anyone exploiting the PHP interpreter by calling it directory, create a separate folder php-cgi-bin. Only the php interpreter will go there. Copy /usr/local/apache/php/bin/php-cgi into new php-cgi-bin directory.

 /usr/local/src/php-5.2.5 root# mkdir  /var/www/php-cgi-bin
 /usr/local/src/php-5.2.5 root# chmod uog+rx  /var/www/php-cgi-bin
 /usr/local/src/php-5.2.5 root# cp /usr/local/apache/php/bin/php-cgi  /var/www/php-cgi-bin
 /usr/local/src/php-5.2.5 root# chmod uog+rx  /var/www/php-cgi-bin/php-cgi

In the Apache configuration file deny all access to the php-cgi-bin directory by using “Deny from all.” Modify the Apache configuration file adding the AddHandler directives. This is similar to what was done in the previous section, except we no longer have the line “LoadModule php5_module modules/libphp5.so”. Instead we are going to tell Apache what action to take with the line “Action application/x-httpd-php /php-cgi-bin/php-cgi”. The following lines get added to /usr/local/apache/conf/httpd.conf

# ScriptAlias: This controls which directories contain server scripts.
ScriptAlias /php-cgi-bin/ "/var/www/php-cgi-bin/"
# Associate file extensions with PHP
AddHandler application/x-httpd-php .php
AddHandler application/x-httpd-php .php3
AddHandler application/x-httpd-php .inc
AddHandler application/x-httpd-php .class
AddHandler application/x-httpd-php .module
# Activates a CGI script for a particular handler or content-type
Action application/x-httpd-php /php-cgi-bin/php-cgi
#
DirectoryIndex index.html index.php
# "/usr/local/apache/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
<Directory "/var/www/php-cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

Test that PHP scripts are now running under the Apache web server.

 /usr/local/src/php-5.2.5 root# echo "<?php phpinfo(); ?>" > /var/www/htdocs/index.php
 /usr/local/src/php-5.2.5 root# chmod uog+rx /var/www/htdocs/index.php
 /usr/local/src/php-5.2.5 root# /usr/local/apache/bin/apachectl start

Access the index.php file by going to “http://127.0.0.1/index.php” with your browser. Once you see PHP is working, stop the web server, remove the index.php info.

 /usr/local/src/php-5.2.5 root# /usr/local/apache/bin/apachectl stop
 /usr/local/src/php-5.2.5 root# rm /var/www/htdocs/index.php

The Apache server can now serve up PHP files. The problem is the PHP is still running as the Apache user. Apache running as a non-root user creates a problem in switching user identities. This is where the use execution wrappers becomes important. To quote from the suEXEC Support page, “The suEXEC feature provides Apache users the ability to run CGI and SSI programs under user IDs different from the user ID of the calling web server. Normally, when a CGI or SSI program executes, it runs as the same user who is running the web server.”

Configuring Apache below, all possible configuration options for suEXEC are included, in addition to the previous configuration options we used for Apache. While this is unnecessary, it is helpful to see what is being set. Please see the suEXEC Support for explanation of the options. In the example below, we will create a specific area, /var/www/htdocs/public_area where we are going to later configure CGIs to run under a certain user. This is just an example and should be adjusted to your environment and needs.

 /usr/local/src/ root# cd /usr/local/src/httpd-2.2.8
 /usr/local/src/httpd-2.2.8 root# make clean
 /usr/local/src/httpd-2.2.8 root# ./configure --prefix=/usr/local/apache \
--enable-rewrite \
--enable-so \
--disable-imap \
--disable-userdir --with-mpm=worker \
--enable-suexec \
--with-suexec-bin=/usr/local/apache/bin/suexec \
--with-suexec-caller=httpd \
--with-suexec-userdir=public_html \
--with-suexec-docroot=/var/www/htdocs \
--with-suexec-uidmin=100 \
--with-suexec-gidmin=100 \
--with-suexec-logfile=/var/www/logs/suexec_log \
--with-suexec-safepath=/usr/local/bin:/usr/bin:bin \
--with-suexec-umask-=022
 /usr/local/src/httpd-2.2.8 root# make
 /usr/local/src/httpd-2.2.8 root# make install
 /usr/local/src/httpd-2.2.8 root# /usr/local/apache/bin/apachectl start

Look in the Apache error log to confirm that suEXEC mechanism is enabled. There will be a log entry in /var/www/logs/error_log:

[notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)

Check the parameters used to compiled suEXEC:

 /usr/local/src/httpd-2.2.8 root# /usr/local/apache/bin/suexec -V

No let us create a script under /var/www/htdocs/public_html/cgi_bin to run as user jbond. We will set this up to run under the virtual host 127.0.0.1. The following is a minimal example of using suEXEC in the Apache configuration file (/usr/local/apache/conf/httpd.conf):

<VirtualHost 127.0.0.1:80>
# Execute all scripts as user as user jbond, group spy
SuexecUserGroup jbond spy

# Maximum 1 CPU second to be used by a process
RLimitCPU 1 1
# Maximum of 25 processes at any one time
RLimitNPROC 25 25
# Allow 10 MB to be used per-process
RLimitMEM 10000000 10000000

<directory /home/jbond/public_html/cgi_bin>
   Options +ExecCGI
  SetHandler cgi-script
</directory>
</VirtualHost 127.0.0.1:80>

Create a simple “hello script” that tells you who is running it in the /var/www/htdocs/public_html/cgi_bin folder. Call the script test.sh

#!/bin/sh
echo "Content-Type: text/html"
echo
echo "Hello world from user <b>`whoami`</b>! "

Adjust the owner and group permissions.

 /usr/local/src/httpd-2.2.8 root# chown -R jbond /var/www/htdocs/public_html
 /usr/local/src/httpd-2.2.8 root# chgrp -R spy /var/www/htdocs/public_html

You can now get the Apache web server to execute the script /var/www/htdocs/public_htm/cgi_bin/test.sh as user jbond by going to “http://127.0.0.1/public_html/cgi_bin/test.sh”.

To provide PHP access is not supported by suEXEC except through the use of Apache module mod_rewrite. A copy of the PHP binary will need to be in each account’s cgi-bin folder. Each account can have its own php.ini file for tailoring to specific account requirements. This will be left as an exercise to the reader, at least until I have time to come back and fill specifics in.

Final Words

In the next post, we will examine running PHP over FastCGI.

In the previous post, “PHP Implementation” we outlined three ways to implement PHP. In this first post, we will examine PHP as an Apache module. When PHP is used as a module, it inherits Apache’s user permissions. PHP basically is operating from within your web server. It starts only once, with the Apache server, loads its settings and extensions only once, and information is stored across sessions. The main advantage is speed.

The configuration, at minimum, will require letting PHP know where the Apache’s apxs tool resides. The apxs tool is a “tool for building and installing extension modules for the Apache HyperText Transfer Protocol (HTTP) server.” In other words, it is the interface between Apache and third-party modules.

For Mac OS X Leopard users, Leopard’s libtool creates php.dSYM instead of the php binary. Gcc calls dsymutil when it creates an executable directly from a source file with debugging enabled so the debugging information will be available. So, under normal libtool operations, .dSYM should only occurs if the object file (php) is temporary. Really old autoconfs fail to detect the correct exe extension and instead of ” set it to ‘.dSYM’. Not really sure what is going on with Leopard, since autoconf is up to date, but I do have a solution. Compile with the CFLAGS set it “-O2.” By doing so, you are not having the compiler turn on the -finline-functions, -funit-at-a-time and -frename-registers options. See GCC Command Options document section “4.10. Options That Control Optimization” for additional details on optimization options. Not sure why Leopard does not behave with higher optimization, but sometimes you have to do what works.

 /usr/local/src root# cd php-5.2.5
 /usr/local/src/php-5.2.5 root#  CC=gcc CFLAGS="-O2" ./configure --with-apxs2=/usr/local/apache/bin/apxs \
--prefix=/usr/local/apache/php

Before doing any installation, we need to modify the Apache configuration file to load PHP. Do this be adding the following lines to the /usr/local/apache/conf/httpd.conf file:

# Load the PHP module
LoadModule php5_module        modules/libphp5.so
# Associate file extensions with PHP
AddHandler application/x-httpd-php .php
AddHandler application/x-httpd-php .php3
AddHandler application/x-httpd-php .inc
AddHandler application/x-httpd-php .class
AddHandler application/x-httpd-php .module
#
DirectoryIndex index.html index.php

At this point, you are ready to make and install PHP. First, you will need to change the permission of the Apache instdso.sh since we were pretty restrictive of permissions in our Apache installation. After installing PHP, we will want to place a version of php.ini in /usr/local/apache/php/lib.

 /usr/local/src/php-5.2.5 root# chmod u+x  /usr/local/apache/build/instdso.sh
 /usr/local/src/php-5.2.5 root# make
 /usr/local/src/php-5.2.5 root# make install
 /usr/local/src/php-5.2.5 root# cp php.ini-recommended /usr/local/apache/php/lib/php.ini
 /usr/local/src/php-5.2.5 root# /usr/local/apache/bin/apachectl configtest

As with Apache, PHP has modules. In PHP documents frequently modules are called functions. Some functions can be considered dangerous and may be disabled. Examine which functions are compiled into PHP. Below is a table of functions that will be compiled in by default:

The posix function may be used for reconnaissance exploits. If it is not required, you should disable it. For later work, we require it and will leave it enabled. For those wishing to disable it, do so through the use of the “–disable-posix” option to configure. The important point is to know what is required and choose both functions and configuration options accordingly. Do not enable everything because you may need it at some point. This is the incorrect thinking that many software packages, and operating systems, fall into. The more code, the more possible vulnerabilities that can be exploited.

We will be building on this implementation of PHP. We have several future requirements:

We add that functionality with the following configuration options:

• –with-gd option will enable PHP to create and manipulate image files with the support of the GD library.
• –with-mysql option will enable PHP access MySQL database servers.
• –enable-sockets option enable socket functions.
• –with-zlib option will enable the use of compression libraries.

Note: Versions of GD older than gd-1.6 support GIF format images, and do not support PNG. Version of GD greater than gd-1.6 and less than gd-2.0.28 support PNG, but not GIF. GIF support was re-enabled in gd-2.0.28. If you get complaints about graphic libraries missing, it is likely due to the version of GD installed.

Look at the modules/functions installed by default with PHP.

 /usr/local/src/php-5.2.5 root# /usr/local/apache/php/bin/php -m
[PHP Modules]
ctype
date
dom
filter
hash
iconv
json
libxml
pcre
PDO
pdo_sqlite
posix
Reflection
session
SimpleXML
SPL
SQLite
standard
tokenizer
xml
xmlreader
xmlwriter
[Zend Modules]

In the previous post, “Introduction to MySQL” MySQL was setup under the /usr/local/mysql directory. This directory will be used in this configuration to add MySQL support to PHP.

If you are compiling under Mac OS X, you should be familiar MacPorts (see posting “MacPorts Under Mac OS X Leopard“). Make sure the required libraries libjpeg, libpng, gd, and zlib are installed. The configuration of PHP requires the locations of these libraries be specified. All these libraries will be located under /opt/local.

There is an issue with Leopards iconv.h file (/usr/include/iconv.h) that can cause the error message “Undefined symbols” as the compiler complains about “_iconv_close, referenced from: _php_iconv_string in iconv.o.” This is caused by multiple iconv.h files on the system. Try moving the Leopard version of the iconv.h file and then linking it to the Macports version.

 /usr/local/src/php-5.2.5 root# mv /usr/include/iconv.h \
/usr/include/iconv.h.leo_orig
 /usr/local/src/php-5.2.5 root# ln -s /opt/local/include/iconv.h \
/usr/include/iconv.h
 /usr/local/src/php-5.2.5 root# make clean
 /usr/local/src/php-5.2.5 root# CC=gcc CFLAGS="-O2 -fno-omit-frame-pointer" \
CXX=gcc CXXFLAGS="-O2 -fno-omit-frame-pointer -felide-constructors \
-fno-exceptions -fno-rtti" ./configure --with-apxs2=/usr/local/apache/bin/apxs \
--prefix=/usr/local/apache/php --enable-sockets --with-gd --with-mysql=/usr/local/mysql \
--with-zlib-dir=/opt/local --with-jpeg-dir=/opt/local --with-png-dir=/opt/local
 /usr/local/src/php-5.2.5 root# make
 /usr/local/src/php-5.2.5 root# make test
 /usr/local/src/php-5.2.5 root# make install

Under Linux, these libraries will be installed as RPMs under the /usr area. The configuration would be slightly different:

 /usr/local/src/php-5.2.5 root# make clean
 /usr/local/src/php-5.2.5 root# CC=gcc CFLAGS="-O3 -fno-omit-frame-pointer" \
CXX=gcc CXXFLAGS="-O3 -fno-omit-frame-pointer -felide-constructors \
-fno-exceptions -fno-rtti" ./configure --with-apxs2=/usr/local/apache/bin/apxs \
--prefix=/usr/local/apache/php --enable-sockets --with-gd --with-mysql=/usr/local/mysql \
--with-zlib-dir=/usr --with-jpeg-dir=/usr --with-png-dir=/usr
 /usr/local/src/php-5.2.5 root# make
 /usr/local/src/php-5.2.5 root# make test
 /usr/local/src/php-5.2.5 root# make install

Copy the PHP configuration file to the expected location and test the Apache configuration file.

 /usr/local/src/php-5.2.5 root# cp php.ini-recommended /usr/local/apache/php/lib/php.ini
 /usr/local/src/php-5.2.5 root# /usr/local/apache/bin/apachectl configtest
 /usr/local/src/php-5.2.5 root# /usr/local/apache/bin/apachectl start

To test if it worked, create a file /var/www/htdocs/index.php containing the lines:

< ?php
phpinfo();
?>

Access the index.php file through your web browser by going to “http://127.0.0.1/index.php” and you should see a bunch of information concerning your PHP setup. Remove the file.

 /usr/local/src/php-5.2.5 root# /usr/local/apache/bin/apachectl stop
 /usr/local/src/php-5.2.5 root# /bin/rm /var/www/htdocs/index.php

The Apache server can now serve up PHP files. Remember, PHP is running with the same file permission as the Apache server. Some important changes need to be made to the php.ini file. Please see the posting, “PHP Configuration Modifications.”

Final Words

In the next post, we will examine configuring PHP as a CGI.

My last post, “An Apache Implementation,” went through the steps of setting up a secure web server. The posting ended with the server having no CGI capabilities. While this might be secure, the usefulness is somewhat limited. In the next few posts we are going to install PHP three different ways. First, as a an Apache module. The second way is using PHP as a CGI. Finally by running PHP over FastCGI. Paul Hudson, author of PHP in a Nutshell maintains a wiki “Practical PHP Programming.” Like his book, the site is a fantastic source of information. Paul has a section titled, “PHP as a CGI or a module?” where he outlines the advantages and disadvantage of both approaches:

Running PHP as a CGI means that you basically tell your web server the location of the PHP executable file, and the server runs that executable, giving it the script you called, each time you visit a page. That means each time you load a page, PHP needs to read php.ini and set its settings, it needs to load all its extensions, and then it needs to start work parsing the script – there is a lot of repeated work.

When you run PHP as a module, PHP literally sits inside your web server – it starts only once, loads its settings and extensions only once, and can also store information across sessions. For example, PHP accelerators rely on PHP being able to save cached data across requests, which is impossible using the CGI version.

The obvious advantage of using PHP as a module is speed – you will see a big speed boost if you convert from CGI to a module. Many people, particularly Windows users, do not realise this, and carry on using the php.exe CGI SAPI, which is a shame – the module is usually three to five times faster.

There is one key advantage to using the CGI version, though, and that is that PHP reads its settings every time you load a page. With PHP running as a module, any changes you make in the php.ini file do not kick in until you restart your web server, which makes the CGI version preferable if you are testing a lot of new settings and want to see instant responses.

One other advantage of running PHP as a CGI is that you can do so with an execution wrapper. PHP as a module will perform all operations as the Apache user. It is a choice between speed verses privilege separation. Running PHP over FastCGI will add speed while keeping privilege separation.

Sources of Information

There are many great sources of information on PHP. There are even sites dedicated to trying to list them. Below are just a few sources that proved most helpful when setting up PHP.

• Paul Hudson, author of PHP in a Nutshell has created the wiki Practical PHP Programming where he has made available all the valuable information from his book.
• Apache Security by Ivan Ristic. I have mentioned before what a great resource Ivan’s book is. From his Apache Security blog, Ivan has made Chapter 3: PHP available to everyone.
• PHP Manual. The best source for most up-to-date information. Pay particular attention to chapters on “Installation and Configuration” and “Security.” The security chapter does have sections on “Installed as CGI binary” and “Installed as an Apache module.”
• FastCGI by Brian Nickel
• Apache + Chroot + FastCGI + PHP FAQ. While I am not going to use Chroot with this implementation, that FAQ is a good resources and discusses additional security measure that are very useful.
• PHP Security Blog by Stefan Esser. Stefan is the developer of Suhosin. While he has not been posting lately, it is good to look over his site. Back in December of 2006, Stefan decided to leave the PHP Security Response Team. His post, “Retired from security@php.net” provides great insight.

Hardened-PHP

Additional security for PHP can be found through the Hardened-PHP project. The project was “founded to protect PHP users and servers against present and future security holes. Therefor we can help you to protect your application and/or your server.” Of particular interest is the project’s development of Suhosin. Suhosin is “an advanced protection system for PHP installations. It was designed to protect servers and users from known and unknown flaws in PHP applications and the PHP core.” Please see the “Feature List” page for a listing of all the additional functionality/protection Suhosin adds to PHP. We are not going to be installing Suhosin today because the latest version of Suhosin is only for PHP 5.1.6. Because of the bug and security fixes, we will be installing PHP 5.2.5.

Installing PHP

Most operating systems have PHP already installed. This post is going to demonstrate three different ways PHP can be installed on a Linux or Mac OS X system.

Download PHP

The first step is to download the latest source code for PHP.

 root# cd /usr/local/src
 /usr/local/src root# wget http://us.php.net/get/php-5.2.5.tar.gz/from/www.php.net/mirror

At this point, check the integrity. Calculate the MD5 sum of the source and compare it to the signature file. Mac OS X users, please note use the command /sbin/md5 instead of md5sum.

 /usr/local/src root# md5sum php-5.2.5.tar.gz
61a0e1661b70760acc77bc4841900b7a  php-5.2.5.tar.gz

Uncompress and untar the sources.

 /usr/local/src root# tar xzf php-5.2.5.tar.gz

Final Words

In the next post, we will examine configuring PHP as a module.