txtdb

the text-file database

GOAL:

txtdb provides a simple, portable, transparent, persistent data store for online and offline application developers. All data is stored in text files that can be independently read or written by other programs.

Basics:

Everything is a string

Everything is append only

Everything is a key/value pair (e.g. pet:dog, name:bob)

Everything is stored in human readable text files

Example:

In Python:

import txtdb

# writing
db = txtdb.open("./data")
db.append("food/fruits", {"name":"apple", "color":"green", "shape":"sphere"})
db.append("food/fruits", {"name":"banana", "color":"yellow"})

# reading
c = db.read("food/fruits", {"color":"green"})
print c.next() # prints {'color': 'green', 'shape': 'sphere', 'name': 'apple'}
print c.next() # prints None

If we type 'find data' on the command line in the same directory we see:

% find ./data
data
data/food
data/food/fruits.txt

And, if we type 'cat data/food/fruits.txt' on the command line we see:

% cat ./data/food/fruits.txt
color:green shape:sphere name:apple
color:yellow name:banana

Details

Unlike traditional database systems that typically force clients to connect to a separate database daemon, txtdb is a library that is linked in with an application. Consistency between different threads and different processes is provided using system-level advisory locks.

txtdb is an append-only database. Yep, no delete. This means that as data changes, those changes are simply appended to the database. For example, if txtdb is used to store user account information then each change (say an updated password) is just appended to the data store. The next time the affected user's credentials are checked, the application reads only that last entry for that user in the database which has the correct data (in our case the updated password).

Terminology:

You append a set of key-value pairs under a given classification. For example, you might append user:bob under account. classifications can be hierarchical so could append ssn:133701337 under account/bob/personal and tel:555-123-4567 under account/bob/public.

Why another database?

SQL is too heavyweight for simple tasks and standard files don't provide the consistency and durability guarantees required for multiple readers and writers. Berkeley DB is simple but not simple enough -- the backend store is not transparent. XML databases are transparent but many have needlessly complicated query syntax.

Given these limitations we define three key requirements for txtdb:

A new user should be able to learn and use txtdb in ~15 minutes.
txtdb should prevent multiple threads or processes from corrupting data.
The txtdb data store must be human readable and editable