Get
Version 0.1:
pytxtdb - Python Module
libtxtdb - C Library

txtdb

the text-file database

GOAL:

txtdb provides a simple, portable, transparent, persistent data store for online and offline application developers. All data is stored in text files that can be independently read or written by other programs.

Basics:
  • Everything is a string
  • Everything is append only
  • Everything is a key/value pair (e.g. pet:dog, name:bob)
  • Everything is stored in human readable text files
  • Example:
    In Python:
    import txtdb
    
    # writing
    db = txtdb.open("./data")
    db.append("food/fruits", {"name":"apple", "color":"green", "shape":"sphere"})
    db.append("food/fruits", {"name":"banana", "color":"yellow"})
    
    # reading
    c = db.read("food/fruits", {"color":"green"})
    print c.next() # prints {'color': 'green', 'shape': 'sphere', 'name': 'apple'}
    print c.next() # prints None
    
    If we type 'find data' on the command line in the same directory we see:
    % find ./data
    data
    data/food
    data/food/fruits.txt
    
    And, if we type 'cat data/food/fruits.txt' on the command line we see:
    % cat ./data/food/fruits.txt
    color:green shape:sphere name:apple
    color:yellow name:banana
    
    Details
    Unlike traditional database systems that typically force clients to connect to a separate database daemon, txtdb is a library that is linked in with an application. Consistency between different threads and different processes is provided using system-level advisory locks.

    txtdb is an append-only database. Yep, no delete. This means that as data changes, those changes are simply appended to the database. For example, if txtdb is used to store user account information then each change (say an updated password) is just appended to the data store. The next time the affected user's credentials are checked, the application reads only that last entry for that user in the database which has the correct data (in our case the updated password).

    Terminology:
    You append a set of key-value pairs under a given classification. For example, you might append user:bob under account. classifications can be hierarchical so could append ssn:133701337 under account/bob/personal and tel:555-123-4567 under account/bob/public.
    Why another database?
    SQL is too heavyweight for simple tasks and standard files don't provide the consistency and durability guarantees required for multiple readers and writers. Berkeley DB is simple but not simple enough -- the backend store is not transparent. XML databases are transparent but many have needlessly complicated query syntax.

    Given these limitations we define three key requirements for txtdb: