Note: The interface IDocStore and its implementations are preview code. We’ve only started playing around with it for production use and will likely introduce changes before we ship any features of MindTouch depending on it. Your feedback is welcome and appreciated.

As part of Dream 1.7 (which is used by MindTouch 9.08), we’ve introduced a schema-less Xml storage inspired by Bret Taylor’s blog post “How FriendFeed uses MySQL to store schema-less data” about a schema-less storage system for JSON blobs used by FriendFeed. Since MindTouch is very much Xml centric and we like the richness of XPath, we decided to adapt the idea to using Xml as the document format.

Why “schema-less” and what does that mean?

In a traditional database, data is split into columns stored in rows in a table. For data more complex than a flat set, other tables are used and referenced from the the first table. That means that to add a new field to your record, you have to create a new column, an operation that can have significant performance implications for that table. In addition, if you choose optimize query performance, you may choose to index that new column, again affecting the the table’s performance during the creation of the index.

But what if you just serialized your record and stuffed that blob into a single column? I mean aside from having just given your DBA an ulcer… Well, you would have the data stored, but in terms of a relational database, you’ve just removed all benefits typically bestowed on a row. How are you going to get this data back out? Pulling each record out and examining it for a match is not a realistic option. Ok, you create an id for the object, and now you have a key/value store on top of an RDBMS… Still lost the ability to query on any other part of your record.

Chances are that you generally only need to retrieve records based on some other field in that record. Adding more columns and populating each for fields to be queried lands you right back where you started. How about creating a self-contained “index” by creating key/value pair tables, one for each field, with values in the rows being taken from your record and linking back to the main storage table. Since each of these index tables is a table of its own index, creation and maintenance no longer affects the main table. You can add an index and populate it without affecting the use of any other key or the data itself.

Introducing IDocStore

This is the storage model that IDocStore follows. The blob record format used is xml, allowing you to store simple fields, or complex hierarchies. Fields to index are defined by using XPath expressions, which makes it possible to not only index single values in the Xml document, but even lists of values. For example, given the following xml:

<user id="123" >
  <name>bob</name>
  <groups xmlns:g="http://foo/groups">
    <group>admin</group>
    <group>contributor</group>
  </groups>
</user>

You could define an index xpath of g:groups/g:group and be able to find bob with a query for either admin or contributor.

The interface of IDocStore is simply this:

public interface IDocStore {
  bool Put(XDoc doc, bool force);
  void Delete(string docId);
  XDoc Get(string docId);
  IList<XDoc> Get(string keyName, string keyValue);
}

We don’t differentiate between Create and Update, simply using Put. We also limit Get to querying on either the primary document id returning at most a single document, or an xpath index, which may return more than one document.

In it’s current incarnation, IDocStore requires the docId already exist in in the document at a configured XPath (default is @id). Lacking a natural key, a Guid could always be used to define the id. Another consideration is to add auto-key generation.

Dream contains two implementations of the IDocStore, MysqlDocStore and SqliteDocStore.

Sqlite Implementation

Starting with Dream 1.7.0 (used by MindTouch 9.08), every service is provided a DataCatalog backed by its own sqlite database (although this can be swapped out for any other database provider via configuration). This can be used as the backend for the SqliteDocStore.

The sqlite implementation, SqliteDocStore, is the simpler of the two. It does not yet include optimistic locking, meaning that Put will force an overwrite of existing data, regardless what force is set to. Also, since the DB file that backs the database cannot be shared by multiple processes, an instance of SqliteDocStore assumes that is the authoritative owner of the storage path, and therefore does index definition and maintenance at instantiation time:

IDocStore store = new SqliteDocStore(_catalog, new XDoc("config")
  .Elem("name", "user")
  .Elem("id-xpath","@id")
  .Start("namespaces")
    .Start("namespace").Attr("prefix","g").Attr("urn","http://foo/groups").End()
  .End()
  .Start("indicies")
    .Start("index").Attr("name","group").Attr("xpath","g:groups/g:group").End()
  .End());

This initialization will either use or create a new store named user in the specified _catalog. It will then perform indicies, adding/removing indicies so that only the index is group.

Mysql Implementation

The mysql implementation, MysqlDocStore,, by virtue of using a remote resource, removes index maintenance from the IDocStore implementation, and adds MysqlDocStoreManager for index manipulation, including updating an index. For this reason, an instance MysqlDocStore requires an instance of MysqlDocStoreManager (really, an instance of an IMysqlDocStoreIndexer) so that it can update the indicies when the documents change.

MysqlDocStore also implements optimistic locking, meaning that the store will attach a revision attribute to any document returned and use that revision to halt an update should the revision in the database be newer (or force is set to true on Put).

Bringing IDocStore to production use

Although an RDBMS is used as the underlying storage, the schema-less store is not ACID compliant. A request based on an index may not result in all the documents that match that index, since the index is maintained asynchronously from the update action. However, the result set it pruned to at least never return a false positive. This is analogous to authoring a page in MindTouch and waiting for the search index to pick up the changes. For most document storage usages, this should be fine.

Schema-less storage is useful when the data to be stored may have unknown or frequently changing structure and simple access patterns, i.e. primary or single key queries, are sufficient. Since MindTouch services often deal with this type of storage and complex DekiScript templates could also benefit from arbitrary storage, we’re experimenting with this as backing store for future extensions, and maybe even expose the storage as a service via an extension so that DekiScript could read from it and JEM could write to it.

If you have use cases for this storage facility, we’d love to hear from you as we test out and mature the design for production. The code is available for preview purposes now in the trunk and 1.7 branches on our source control server, and is part of the binary distribution going out with mindtouch.dream.dll in MindTouch 9.08