I spent the past weekend traveling. When I travel, I listen to podcasts. Traveling is my time to catch up on some great content. I’ll post more on that later. I wanted to draw attention to FLOSS Weekly Episode #26 interview with D. Richard Hipp, creator and lead developer of SQLite. Randal Schwartz and Leo Laporte always do a great job with these interviews. What is so interesting about SQLite? Take a look at the features:

• Transactions are atomic, consistent, isolated, and durable (ACID) even after system crashes and power failures.
• Zero-configuration – no setup or administration needed.
• Implements most of SQL92. (Features not supported)
• A complete database is stored in a single cross-platform disk file.
• Supports terabyte-sized databases and gigabyte-sized strings and blobs. (See limits.html.)
• Small code footprint: less than 250KiB fully configured or less than 150KiB with optional features omitted.
• Faster than popular client/server database engines for most common operations.
• Simple, easy to use API.
• Written in ANSI-C. TCL bindings included. Bindings for dozens of other languages available separately.
• Well-commented source code with over 99% statement test coverage.
• Available as a single ANSI-C source-code file that you can easily drop into another project.
• Self-contained: no external dependencies.
• Cross-platform: Linux (unix), MacOSX, OS/2, Win32 and WinCE are supported out of the box. Easy to port to other systems.
• Sources are in the public domain. Use for any purpose.
• Comes with a standalone command-line interface (CLI) client that can be used to administer SQLite databases.

Those features should be enough to make one take notice. SQLite is also small, compact, portable, efficient, and serverless. It is designed so it can be plugged directly into programs, scripts, or web applications. This provides programs with a lightweight relational database engine that has no external dependencies.

SQLite is very different from MySQL and PostgreSQL. Yet, frequently developers can end up using a full fledge database when something much smaller and effecient could be used. Unfortunately, there are no current comparisons in performance to MySQL and PostgreSQL. The page off the SQLite site, “Database Speed Comparison” does state that the document “describes a speed comparison between an older version of SQLite against archaic versions of MySQL and PostgreSQL.” Still, at least in the past for some operations, SQLite demonstrated impressive speeds compared to PostgreSQL and MySQL:

• SQLite 2.7.6 is significantly faster (sometimes as much as 10 or 20 times faster) than the default PostgreSQL 7.1.3 installation on RedHat 7.2 for most common operations.
• SQLite 2.7.6 is often faster (sometimes more than twice as fast) than MySQL 3.23.41 for most common operations.

The list of folks using SQLite is impressive: Google Gears, Firefox’s mozStorage, Apple (Safari, Mail, Core Data, Aperture), smf framework in Solaris 10 is using SQLite as its data store, PHP, yum, monotone, AOL email client, Skype, McAfee, along with many additional companies. There are extensions allowing SQLite to be used with languages such as Perl, Python, Ruby, PHP, Java, TCL, .NET, Smalltalk, and many other languages. SQLite compiles and runs on Windows, Linux, Mac OS X, BSD, Solaris, AIX, HP-UX, Symbian, WinCE, VX Works, OS/2, and the NetBSD toaster. SQLite databases are binary compatible, which means they work natively on all systems without any need for conversion. At this point, you are probably beginning to understand why SQLite is so interesting.

Richard has done a talk over at Google TechTalks that provides a good overview of SQLite.

SQLite is made to be easy to setup and use. If the above information has made you somewhat interested, the below instruction on how to setup SQLite should help get you started.

Installation

Each operating system will be somewhat different when it comes to the binary installation. While there will be different filenames, the idea and ease of installation is the same across OSs. For example, there are two files for use under Mac OS X (see the SQLite site for the most recent files):

Installation can be done by using the binaries supplied from the SQLite site. Generally, you do not need to install SQLite on its own. It will either comes installed with the OS or extensions to programming languages will come with SQLite. This is the advantage of being so small. It is easy to include SQLite.

Below is an example of how to install SQLite binaries under Mac OS X. Mac OS X does come with SQLite installed (sqlite3) by default.

root# cd /usr/local/src /usr/local/src root# mkdir SQLite /usr/local/src root# cd SQLite /usr/local/src/SQLite root# wget http://www.sqlite.org/sqlite3-3.5.7-osx-x86.bin.gz /usr/local/src/SQLite root# wget http://www.sqlite.org/sqlite3_analyzer-3.5.4-osx-x86.bin.gz /usr/local/src/SQLite root# gunzip sqlite3-3.5.7-osx-x86.bin.gz /usr/local/src/SQLite root# gunzip sqlite3_analyzer-3.5.4-osx-x86.bin.gz /usr/local/src/SQLite root# chmod u+x sqlite3-3.5.7-osx-x86.bin /usr/local/src/SQLite root# chmod u+x sqlite3_analyzer-3.5.4-osx-x86.bin /usr/local/src/SQLite root# ./sqlite3-3.5.7-osx-x86.bin SQLite version 3.5.7 Enter ".help" for instructions sqlite>

Below are instructions for installation via source code, which would be applicable for non-windows OSs:

root# cd /usr/local/src /usr/local/src root# mkdir SQLite /usr/local/src root# cd SQLite /usr/local/src/SQLite root# wget http://www.sqlite.org/sqlite-amalgamation-3.5.7.tar.gz /usr/local/src/SQLite root# tar xzf sqlite-amalgamation-3.5.7.tar.gz /usr/local/src/SQLite root# cd sqlite-3.5.7 /usr/local/src/SQLite/sqlite-3.5.7 root# ./configure /usr/local/src/SQLite/sqlite-3.5.7 root# make /usr/local/src/SQLite/sqlite-3.5.7 root# make install

SQLite and Perl

To provide an example of how to use SQLite below are instructions on installing and using SQLite with the Perl language. As previously discussed, SQLite can be used with many languages. Perl was chosen in honor of Randal Schwartz. While Randal can probably program in all the languages listed above, many first became aware of Randal through Perl. You will find SQLite is just as easy to install and use with your favorite language.

The SQLite extension for Perl contains its own version of SQLite. There really is no need to compile and install SQLite beforehand. While SQLite is binary compatible, different version of the database may not be compatible. The Perl module DBD::SQLite uses an old SQLite database format. DBD::SQLite::Amalgamation uses the most most recent SQLite database format. If you get the error message “SQL error: file is encrypted or is not a database,” this might be caused by different database versions.

To install DBI and DBD::SQLite::Amalgamation using CPAN:

 root#  perl -MCPAN -e shell
   cpan> install DBI
   cpan> install DBD::SQLite::Amalgamation

To install DBI and DBD::SQLite::Amalgamation using source code.

root# cd /usr/local/src /usr/local/src root# mkdir perl /usr/local/src root# cd perl /usr/local/src/perl root# wget http://search.cpan.org/CPAN/authors/id/T/TI/TIMB/DBI-1.604.tar.gz /usr/local/src/perl root# wget \ http://search.cpan.org/CPAN/authors/id/A/AU/AUDREYT/DBD-SQLite-Amalgamation-3.5.6.tar.gz /usr/local/src/perl root# tar xzf DBI-1.604.tar.gz /usr/local/src/perl root# tar xzf DBD-SQLite-Amalgamation-3.5.7.tar.gz /usr/local/src/perl root# cd DBI-1.604 /usr/local/src/perl/DBI-1.604 root# perl Makefile.PL /usr/local/src/perl/DBI-1.604 root# make /usr/local/src/perl/DBI-1.604 root# make test /usr/local/src/perl/DBI-1.604 root# make install /usr/local/src/perl/DBI-1.604 root# cd ../DBD-SQLite-Amalgamation-3.5.7 /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# perl Makefile.PL /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# make /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# make test /usr/local/src/perl/DBD-SQLite-Amalgamation-3.5.7 root# make install

Create Database

To create a database sample.db, issue the commands:

 root# cd /usr/local/code
 /usr/local/code root# /usr/local/bin/sqlite3 sample.db
SQLite version 3.5.7
Enter ".help" for instructions
sqlite> .quit

Create Table

Creating the table “event” can be done via the command line:

 /usr/local/code root# /usr/local/bin/sqlite3 sample.db "create table event (id INTEGER
                  PRIMARY KEY,odate DATE, description TEXT);"

Inserting Data Into the Table

To insert data into the table via command line:

 /usr/local/code root# /usr/local/bin/sqlite3 sample.db "insert into event (id, odate, description)
       values (1,'2008-04-03 17:59:26','Created entry into SQLite event table.');"

Retrieve the Data From the Table

To retrieve the information via command line:

 /usr/local/code root# /usr/local/bin/sqlite3 sample.db "select id, odate, description from event;"

Creating Database, Insert Data, Retrieve Records via Perl

Below is a Perl program that will create and enter data using DBI DBD::SQLite:

#!/usr/bin/perl

   use DBI;

   # Connect and create database if it does not already exist
   $dbh = DBI->connect( "dbi:SQLite:data.dbl" ) || die "Cannot connect: $DBI::errstr";

   # Create table
   $dbh->do( "CREATE TABLE authors ( lastname, firstname )" );
   $dbh->do( "CREATE TABLE books ( title, author )" );

   # Insert into tables
   $dbh->do( "INSERT INTO authors VALUES ( 'Conway', 'Damian' ) " );
   $dbh->do( "INSERT INTO authors VALUES ( 'Booch', 'Grady' ) " );
   $dbh->do( "INSERT INTO books VALUES ( 'Object Oriented Perl', 'Conway' ) " );
   $dbh->do( "INSERT INTO books VALUES ( 'Object-Oriented Analysis and Design',
                                             'Booch' ) ");
   $dbh->do( "INSERT INTO books VALUES ( 'Object Solutions', 'Booch' ) " );

   # Display data from tables
   $sth = $dbh->prepare( q( SELECT a.lastname, a.firstname, b.title
                                           FROM books b, authors a
                                           WHERE b.title like '%Orient%'
                                       AND a.lastname = b.author ) );
   $rc = $sth->execute();
   if ($rc) {
     while (my($lastname,$firstname,$title) = $sth->fetchrow_array()) {
        print "Name: $lastname, $firstname\nTitle: $title\n";
     }
   }
   else {
     print "Problem with SELECT statement: SELECT a.lastname, a.firstname, b.title
                FROM books b, authors a WHERE b.title like '%Orient%' AND a.lastname = b.author\n";
   }

   # Disconnect from database.
   $dbh->disconnect;

Please note that there is a know issue between DBI and SQLite where a warning message “closing dbh with active statement handles” might be generated. For now, there is no resolution. The code does work. It is only a warning message involving closing the database. You may want to keep an eye open for future resolution.

Additional Information

Mike Chirico has done a nice tutorial on using SQLite, titled “SQLite Tutorial.” While this posting has used a few examples to demonstrate how to create a database, create a table, insert values into the table, and read values from the table, please view Mike’s tutorial for additional commands and more in-depth explanations.

Mike Owens has written a really good book on SQLite, titled “The Definitive Guide to SQLite.” Mike has also made available his presentation for OSCON titled “Programming with SQLite.” The presentation covers “SQLite’s design, operation, capabilities, and limitations, providing developers with a better idea of how, when, and where to best put it to use in their applications.”

Conclusions

There are many more tools and much information available on SQLite. Firefox even has a add-on, SQLite Manager, that allows you to manage SQLite database on your computer. The truth is, I am not sure where I am going to use SQLite. There are plenty of places where I am now thinking I should be using SQLite. I know that I am glad to have it as a tool that I can use. I hope this introduction has captured your interest. Thanks to Randal Schwartz and Leo Laporte for doing the FLOSS Weekly podcast and making me aware of this valuable tool. A special thanks to D. Richard Hipp and Dan Kennedy for developing such a powerful tool.