The Database

A few diagrams of the RT’s database schema are available:

• Steve Turner’s Model
• The official RT 3.4 schema
• Jesse Vincent’s model (pdf)

If you are unfamiliar with some of the keys and conventions used in Visio graphs, the below table provides some helpful information. Mandatory (not null) columns are displayed in bold.

PK	Primary key	–––0+	0 or 1
FK	Foreign key	–––0<–	0 or more
U	Unique column	–––++	Exactly 1
I	Indexed column	–––+<–	1 or more
O	Optimal columns

Further clarification can be found in chapter 8 titled “Architect” of the “RT Essentials” book from O’Reilly. Look for the section “Logical and Object Model” which takes a tour of RT’s logical and object models.

Perl Module

Referring once more the the “RT Essentials” book’s chapter 8 on architecture, the below diagram maps the layers involved with RT.

A quick overview of what the provide:

1. Allowing a database independent interface to Perl is the DBI module.
2. DBIx::SearchBuilder encapsulate SQL queries and rows in simple perl objects allowing object-oriented applications like RT to talk to a table-oriented relational database.
3. The RT application platform libraries provide database connectivity, logginng infrastructures, users, groups, access control, links, etc. Basically it is the guts of RT.
4. The RT ticketing system libraries uses the RT application platform.
5. The Mason handler run on top of the RT core libraries and provides a wrapper around the Mason templating system. The Mason templates consists of the user interface templates, which designed for end users to interact with their browsers, and the REST templates, which are designed to be easy for other software to interact with RT.

Creating, querying, and editing tickets in an RT instance could be done by using RT Command Line Interface (CLI) calls embedded in programs. Or, one could directly plug into the RT libraries. To maintain compatibility with future releases of RT, we will be using RT’s built in REST interface.

Fortunately, Dmitri Tikhonov has created the RT::Client::REST. If Ruby is your preferred language, Tom Lahti has cared a Ruby library to interface with RT.

Jesse Vincent posted recently on the Best Practical blog, “RT 4 – status report.” While it will be awhile before RT4 is out, Jesse has written that “RT4 is based on Jifty and serves up both the legacy /REST/1.0 interface and Jifty’s much more modern REST interface.” End result is that compatibility will be maintained.

Perl Modules Installation

Make sure to follow the instructions from “Request Tracker Installation (Part 2 of 2).” The Perl foundation defines kwalitee as “a set of formalities that tend to coincide with quality, according to consensus. It is of course much less useful than quality, but at least it can be measured.” Install the Perl module Module::CPANTS::Analyse and Test::Kwalitee, along with supporting modules, for quality testing.

/usr/local/src root# perl -MCPAN -e 'install Module::CPANTS::Analyse' /usr/local/src root# perl -MCPAN -e 'install Test::Kwalitee'

If you have read any of Terry Goodkind’s Sword of Truth series of books, you maybe familiar with his character Zedd saying, “If the road is easy, you’re likely going the wrong way.” No where is this more true than in IT. Life is made a bit easier if you check active bugs when setting up software. Fortunately, RT-Client-REST does have an active bug listing.

There is a bug involving CustomFields change in RT 3.8 and how RT matched on the # symbol. RT incorrectly matched when using the REST interface because RT::Client::REST had a CustomField with a # at the end. Jerrad Pierce has just posted that the necessary changes were mote extensive and the code should be pulled down from SVN. We will pull the code from there.

/usr/local/src root# svn checkout \
     http://rt-client-rest.googlecode.com/svn/trunk/ rt-client-rest-read-only
/usr/local/src root# cd  rt-client-rest-read-only/rt-client-rest
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# perl Makefile.PL
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# make
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# make test
/usr/local/src/rt-client-rest-read-only/rt-client-rest root# make install

Connecting up through SSL requires a few additional steps. make sure to install the Perl module Crypt::SSLeay.

 /usr/local/src root# perl -MCPAN -e 'install Crypt::SSLeay'

Ruby

If you need to install Ruby please see my earlier post “Implementing Puppet: Act One.” Ruby 1.9.x is a fairly significant change. See Josh Haberman post “Ruby 1.9.1 released,” Markus Prinz post “Ruby 1.9 – What’s new? What’s changed?“, and Peter Cooper’s post “23 Useful Ruby 1.9 Links and Resources.” If you are working with Ruby, you need Dave Thomas‘ book “Programming Ruby 1.9: The Pragmatic Programmers’ Guide,” which is about to be released and is available in electronic format.

When you issue the “gem install rt-client,” errors involving the TMail file tmailscanner.c will occur. First, it was looking for header files in directory /usr/local/include/ruby-1.9.1 instead of /usr/local/include/ruby-1.9.1/ruby and complaining about “re.h: No such file or directory“. As Zedd would say, “Nothing is ever easy.” If you fix that problem, TMail will complain about “struct RString.” This is a show stopper if you want to use Ruby 1.9.x. When this gets fixed, I will try and come back and update this post.

Connecting Securely

Modify the RT_SiteConfig.pm to use port 443.

Before (Without SSL): Set($WebBaseURL , " http://rt.yourdomain.com");
After (With SSL): Set($WebBaseURL , " https://rt.yourdomain.com:443");

Please change rt.yourdomain.com to the appropriate host value for your organization.

The REST Interface does not support HTTP-Authentication. If your web server requires users to log in, you will end up with authentication problems. As of this writing, there are problems when when both authentication mechanisms are used together. A work around, if your program is running on the same machine as the web server, is to setup a virtual host for 127.0.0.1 that does not use HTTP-Authentication. Make sure to connections are allowed from client 127.0.0.1 only. The outside world interface can continue to be forced to use HTTPS and HTTP-Authentication.

Sample Program

With the supporting software in place, we can now write a program. Below is a simple program that connects up to OpenSSL’s RT site, pulls out all new and open tickets belonging to the OpenSSL-Bugs queue, and prints out the id, subject, owner, status, and when the ticket was created.

#!/usr/local/bin/perl -w

  use strict;
  use Error qw(:try);
  use RT::Client::REST;
  use Data::Dumper;

  my %Config = (
      server      => 'http://rt.openssl.org/',
      username    => 'guest',
      password    => 'guest',
      queue       => 'OpenSSL-Bugs'
  );
  my $rt = RT::Client::REST->new(
    server => $Config{server},
    timeout => 30,
  );

  try {
    $rt->login(username => $Config{username}, password => $Config{password} );
  }
  catch Exception::Class::Base with {
    die "problem logging in: ", shift->message;
  };

  my @ids;
  try {
    @ids = $rt->search(
        type    => 'ticket',
        query   => qq[
            (Status = 'new' or Status = 'open')
            and
            Queue = '$Config{queue}'
        ],
    );
  }
  catch Exception::Class::Base with {
    die "search failed", shift->message;
  };
  for my $id (@ids) {
    my $ticket = $rt->show(type => 'ticket', id => $id);
    print "ID: $id\n";
    print Dumper($ticket);
   }

Final Thoughts

One of the more difficult aspect of connecting several different open source projects is what to do with various versions of the software. While it is unfortunate that we will have to wait for software to get updated on the Ruby side, we are now ready to start working with Perl. In the next post, we will go through the steps to take a program that monitors activity and informs administrators via email, to a system that uses RT to perform this function. By doing so, we gain operational tracking capability. We are about to start having some fun.

Michael Coté, an industry analyst with RedMonk, interviewed Reductive Lab’s Luke Kaines and Google’s Nigel Kersten on the podcast “Puppet at Google – RedMonk Radio Episode 48.” To quote Coté, “Nigel has been using Puppet to manage ‘many, many thousands’ of Mac desktops used at Google by developers and others. He tells us how he got involved in using Puppet during WWDC last year and quickly applied its use to managing Google Mac desktop.” Pat Eyler also posted, on the On Ruby blog, an interview with James Turnbull. Turnbull is the author of “Pulling Strings with Puppet.” To quote Turnbull on Puppet, “It takes all the pain out of systematically and efficiently managing your host.”

Phil Windley, Founder and Chief Technology Officer (CTO) of Kynetx has been providing some great information on Puppet. Phil recently attended the O’Reilly Velocity 08 conference where he attended a talk given by Luke Kanies. Phil shares his impression in the post “Velocity 08: Puppet In-Depth and Hands-On.” Phil is also the host of IT Conversations’ Technometria podcast. He must have liked what he heard at Velocity 08, for he had Luke Kanies on his show to discuss Puppet. Plus, Phil has been sharing his experience implementing Puppet in his posts “Puppet Fun” and “Using a Pre-Commit Hook to Check Puppet Syntax.”

The point of today’s post is not to discuss how useful Puppet can be. Its worth can be best determined through examining how it is used. I have provided a few links to people making good use of Puppet. I will also provide follow up posts describing integrating Puppet with other security software we have been discussing. Today’s post is simply to set the groundwork by focusing on implementing a basic Puppet setup.

Prerequisites

Most of what Puppet requires will come as part of the Ruby standard library. It is best to use the Ruby that comes with your OS. I will still go through the installation of Ruby. The only other required package is Reductive Lab’s Facter, which is a cross-platform Ruby library for retrieving facts from operating systems.

• Ruby
• Facter

The Ruby standard libraries that are required by Puppet include:

• base64
• cgi
• digest/md5
• etc
• fileutils
• ipaddr
• openssl
• strscan
• syslog
• uri
• webrick
• webrick/https
• rdoc
• xmlrpc

Installation

In order to install all the required packages, we will start by installing Ruby, followed by checking that the required libraries are installed, and finish with Facter. If any of these software packages are installed by default on your OS, you will likely want to skip the corresponding installation steps.

Ruby

The Ruby website describes Ruby as “a dynamic, open source programming language with a focus on simplicity and productivity. It has an elegant syntax that is natural to read and easy to write.” Below are the steps to do a source code installation. Please note that there are other options listed in the Download Ruby page. I will be going through installation using the source code only because there are some systems that do not have Ruby installed. If you are going to be working with Ruby, you may consider purchasing a great reference on Ruby titled “Programmers’ Guide, 3rd Edition” by Dave Thomas, with Chad Fowler and Andy Hunt.

 root# cd /usr/local/src
 /usr/local/src root# wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.7-p72.tar.gz
 /usr/local/src root#  /usr/bin/md5sum ruby-1.8.7-p72.tar.gz
5e5b7189674b3a7f69401284f6a7a36d  ruby-1.8.7-p72.tar.gz

 /usr/local/src root# tar xzf ruby-1.8.7-p72.tar.gz
 /usr/local/src root# cd ruby-1.8.7-p72
 /usr/local/src/ruby-1.8.7-p72 root# ./configure
 /usr/local/src/ruby-1.8.7-p72 root# make
 /usr/local/src/ruby-1.8.7-p72 root# make check
 /usr/local/src/ruby-1.8.7-p72 root# make install

RubyGems

RubyGems is the premier Ruby packaging system. Quoting the RubyGems website, “it provides a standard format for distributing Ruby programs and libraries, an easy to use tool for managing the installation of gem packages, and a gem server utility for serving gems from any machine where RubyGems is installed.” Using RubyGems makes handling libraries a great deal easier. If you are behind a proxy server, you will either need to set HTTP_PROXY or include the proxy setting in the gem command.

 root# cd /usr/local/src
 /usr/local/src root# wget http://rubyforge.org/frs/download.php/43985/rubygems-1.3.0.tgz
 /usr/local/src root# tar xzf rubygems-1.3.0.tgz
 /usr/local/src root# cd rubygems-1.3.0
 /usr/local/src/rubygems-1.3.0 root# ruby setup.rb
 /usr/local/src/rubygems-1.3.0 root# gem --version
1.3.0
 /usr/local/src/rubygems-1.3.0 root# gem update --system
Updating RubyGems
Updating rubygems-update

Ruby Libraries

Check that the libraries are installed with the commands:

 root# ruby -rbase64 -e "puts :installed"
 root# ruby -rcgi -e "puts :installed"
 root# ruby -rdigest -e "puts :installed"
 root# ruby -retc -e "puts :installed"
 root# ruby -rfileutils -e "puts :installed"
 root# ruby -ripaddr -e "puts :installed"
 root# ruby -ropenssl -e "puts :installed"
 root# ruby -rstrscan -e "puts :installed"
 root# ruby -rsyslog -e "puts :installed"
 root# ruby -ruri -e "puts :installed"
 root# ruby -rwebrick -e "puts :installed"
 root# ruby -rrdoc/usage -e "puts :installed"
 root# ruby -rxmlrpc -e "puts :installed"
ruby: no such file to load -- xmlrpc (LoadError)

With RubyGem installed, checking and installing Ruby libraries is fairly easy. For future reference, a gem is a packaged Ruby application or library. Below we are going to look for a xmlrpc packaged library globally and install it locally.

bash-3.00# gem list -b ^xmlrpc
*** REMOTE GEMS ***
xmlrpcs (0.1.3)

bash-3.00#  gem install xmlrpcs
Successfully installed xmlrpcs-0.1.3
1 gem installed
Installing ri documentation for xmlrpcs-0.1.3...
Installing RDoc documentation for xmlrpcs-0.1.3...

Facter

Reductive Labs describes Facter as “a cross-platform Ruby library for retrieving facts from operating systems. Supports multiple resolution mechanisms, any of which can be restricted to working only on certain operating systems or environments. Facter is especially useful for retrieving things like operating system names, IP addresses, MAC addresses, and SSH keys.” To install and confirm it is installed, issue the following commands:

 root# cd /usr/local/src
 /usr/local/src root# wget http://www.reductivelabs.com/downloads/facter/facter-latest.tgz
 /usr/local/src root# tar xzf facter-latest.tgz
 /usr/local/src root# cd facter-1.5.2
 /usr/local/src/facter-1.5.2 root# ruby install.rb
 /usr/local/src/facter-1.5.2 root# facter --version
1.5.2

There is an alternative method for installing Facter using RubyGem. Currently, it does not appear to be working. I’ll include the instructions with the hope that it will be fixed in the future.

 root# gem install --remote --source http://reductivelabs.com/downloads/ facter
 root# facter --version
1.5.2

Puppet

Puppet can be installed via source, packages, or using RubyGem. I’ll demonstrate installation using source and as a gem. Since package installation is dependent on the OS, readers should view the Reductive Labs Wiki.

The source contains both the Puppet server and client code. Download and install Puppet with the following commands:

 root# cd /usr/local/src
 /usr/local/src root# wget http://reductivelabs.com/downloads/puppet/puppet-latest.tgz
 /usr/local/src root# tar xzf puppet-latest.tgz
 /usr/local/src root# cd puppet-0.24.5
 /usr/local/src/puppet-0.24.5 root# ruby install.rb
 /usr/local/src/puppet-0.24.5 root# puppet --version
0.24.5

As with Facter, using RubyGem to install Puppet does not seem to work. I’ll include the instructions and hopefully it will be fixed in the future.

 root# gem install --remote --source http://reductivelabs.com/downloads/ puppet
 root# puppet --version
0.24.5

The Puppet Master

Most of the functionality of Puppet can be run with default configuration. The two things that need to be assigned is a user and a group to run along with a basic configuration to apply to the client.

User and Group

Check if the puppet user and group exist on your system.

 root# id puppet
uid=477(puppet) gid=149(puppet) groups=149(puppet)

If that does not work for your OS, you can grep through the password and group files.

 root# grep puppet /etc/passwd
 root# grep puppet /etc/group

If you need to add the user and group puppet, under most unix version you would do so with the command:

 root# groupadd puppet
 root# useradd -M -g puppet puppet

Mac OS X Leopard (10.5) makes things a little more complicated if you are going to use the command line. You will first need to look at what group ids (gid) are used, choose an unused gid, and then create the puppet group using that gid. This would be accomplished with the commands:

 root# dscl . list /groups PrimaryGroupID | sort -k 2,2 -n | tail -2
rt                 503
httpd            10650
 root# dscl . create /groups/puppet gid 504
 root# dscl . create /groups/puppet passwd '*'
 root# dscl . read /groups/puppet
 root# dscl . list /users UniqueID | sort -k 2,2 -n | tail -2
ossecr             505
httpd              47008
 root# dscl localhost -create /Local/Default/Users/puppet
 root# dscl localhost -create /Local/Default/Users/puppet RecordName puppet
 root# dscl localhost -create /Local/Default/Users/puppet UserShell /sbin/nologin
 root# dscl localhost -create /Local/Default/Users/puppet RealName "Puppet"
 root# dscl localhost -create /Local/Default/Users/puppet UniqueID 506
 root# dscl localhost -create /Local/Default/Users/puppet PrimaryGroupID 504
 root# dscl localhost -create /Local/Default/Users/puppet NFSHomeDirectory /dev/null
 root# dscl . read /users/puppet
AppleMetaNodeLocation: /Local/Default
GeneratedUID: B71B9C49-5C59-4B17-B69C-415589832976
NFSHomeDirectory: /dev/null
PrimaryGroupID: 504
RealName: Puppet
RecordName: puppet
RecordType: dsRecTypeNative:users
UniqueID: 506
UserShell: /sbin/nologin
 root# id puppet
uid=506(puppet) gid=504(puppet) groups=504(puppet)

Configuration File

We are going to create a default configuration file /etc/puppet/manifests/site.pp.

 root# mkdir -p /etc/puppet/manifests
 root# vi /etc/puppet/manifests/site.pp

For a very simple example, we will have Puppet check the user and group ownership of /etc/passwd as well as its permissions. The following would go into site.pp:

file { "/etc/passwd":
  owner => "root",
  group => "bin",
  mode => 644,
}

Starting

With a simple configuration file create, we can now start the master daemon.

 root# puppetmasterd --verbose --no-daemonize
info: Starting server for Puppet version 0.24.5
info: Listening on port 8140
notice: Starting Puppet server version 0.24.5

The –verbose option will turn on verbose logging while the –no-daemonize option forces the daemon to run in the foreground. The daemon will start on TCP port 8140.

The Client

The Puppet client will run as root in order to perform the required configuration actions on the node. When you start the client for the first time, it will generate a local self-signed certificate, connect to the specified master, and request the certificate be signed. Puppet relies on SSL when communicating between the server and clients. Once the certificate is signed, the node will request whatever configuration is specified for that node. Clients running on the same host as the server have their certificates automatically signed.

To test Puppet, have the group permission of the /etc/passwd file be “root” and see how Puppet handles it on the client (example: puppet1.securitymonks.com). The client will first require the server (example: puppetmaster.securitymonks.com) to sign its certificate. The client connects to the server when the puppetd command is run. Until the certificate is signed, the group permission on /etc/passwd will not change. From the client, issue the commands:

 root# chgrp root /etc/passwd
 root# ls -l /etc/passwd
-rw-r--r--  1 root root 2800 Sep 28 05:53 /etc/passwd
 root# puppetd --server puppetmaster.securitymonks.com --verbose --waitforcert 60
notice: Starting Puppet client version 0.24.5
info: No classes to store
info: Caching catalog at /var/puppet/state/localconfig.yaml
 root# ls -l /etc/passwd
-rw-r--r--  1 root root 2800 Sep 28 05:53 /etc/passwd

The –server option tells the client the name of the server (puppetmaster.securitymonks.com). The –waitforcert tells the client to check every 60 seconds for the server to return signed certificate. Over on the server, we see the message:

info: Caching node for puppet1.securitymonks.com
notice: Compiled catalog for puppet1.securitymonks.com in 0.04 seconds

Certificates

The Puppet master will sign certificates using the puppetca command, which controls the Puppet Certificate Authority. While automatically signing certificates is possible, in the interest of security we will be manually approving requests. Please read more about certificate management and security at Reductive Labs’ Wiki.

From the server, puppetmaster.securitymonks.com, list the waiting certificate and sign the witht he commands:

 root# puppetca --list
puppet1.securitymonks.com
 root# puppetca --sign puppet1.securitymonks.com
Signed puppet1.securitymonks.com

Go back to the client and notice that it now has a certificate. Also examine the group permission on the /etc/passwd file.

notice: Got signed certificate
notice: Starting catalog run
notice: //File[/etc/passwd]/group: group changed 'root' to 'bin'
notice: Finished catalog run in 0.08 seconds
 root# ls -l /etc/passwd
-rw-r--r--  1 root bin 2800 Sep 28 05:53 /etc/passwd

Curtain Drops

This brings to a close Act One of this Puppet implementation. A very simple Puppet master daemon with one node has been created. In Act Two, we will do a little more advance configuration. If you plan on working with Puppet, buy the book “Pulling Strings with Puppet: Configuration Management Made Easy” by James Turnbull. It is available in print or as a PDF and is the most extensive source of information in print on Puppet. The Reductive Labs Wiki is another great source for up-to-date information. Of course, the ultimate source for information on Puppet are the folks who created it. Consider hiring Reductive Labs to provide training, services, or support.

While I work on writing Act Two, in order to keep you entertained, please enjoy Luke Kanies’ Puppet presentation given at Silicon Valley’s BayLISA event on August 2006. It is a most informative and entertaining video with a run time of about 90 minutes. Enjoy. Exit stage right.

Normally, I am pleased with myself if I can complete one post a week. Still, I had to put in that extra effort this week in order to get James Turner comic up. The cartoon is bound to become a classic. Well, maybe not among the masses. Okay, maybe just among an elite group of people who can identify languages by the animals on their O’Reilly book covers. Still, that is a pretty special group of people.

Since I am posting a comic strip involving Ruby and Perl, I figured I would add a few pointers of interest. If you are an old time Perl programmer, you will want to check out Jonathan Scott Duff’s posting on, “Everyday Perl 6.” Perlcast, a podcast focus primarily on the Perl programming language, has posted a podcast on “Learning Perl 6.” The presentation was done by Brian D Foy at the Nordic Perl Workshop 2007. Slides along with the audio podcast of the presentation are available.

Not to show favoritism, on the Google Code Blog, they did their fourth podcast where Mark Limber talks on Google SketchUp. To quote the Google SketchUp site:

Developed for the conceptual stages of design, Google SketchUp is a powerful yet easy-to-learn 3D software tool that combines a simple, yet robust tool-set with an intelligent drawing system that streamlines and simplifies 3D design. From simple to complex, conceptual to realistic, Google SketchUp enables you to build and modify 3D models quickly and easily. If you use Google Earth, Google SketchUp allows you to place your models using real-world coordinates and share them with the world using the Google 3D Warehouse.

Ruby is the scripting language that is used in SketchUp. Sorry Perl.