Custom Extensions
It's extensible.
The key/value architecture of YapDatabase is just the foundation for an advanced plugin system. These plugins are called "extensions". YapDatabase ships with a number of helpful extensions. And, naturally, you can write your own.
Extensions are integrated deep into the architecture:
- they have access to sqlite internals
- they can create their own dedicated sqlite tables
- they take part in transactions
- they implement hook methods which are called whenever items are inserted, modified, removed, etc
In order to implement your own custom extension, you'll be overriding 3 base classes:
All the methods that you need to override are private / internal. So they are listed & documented in the implementation files (.m not .h).
The "meat & potatoes" of extensions are the hook methods. So it's instructive to start by listing & discussing these methods first, and then building up on top of this foundation.
If you consider the key/value nature of YapDatabase, the ReadWrite API is rather sparse. There is a method to set/update an object ([transaction setObject:forKey:inCollection:]). And a few methods to remove objects (such as [transaction removeObjectForKey:inCollection:]). And that's mostly it. This simplicity makes YapDatabase easy to use, and also simplifies life for the extension developer.
So let's start with the 2 most basic hook methods:
/**
* YapDatabaseReadWriteTransaction Hook, invoked post-op.
* Corresponds to [transaction setObject:object forKey:key inCollection:collection] &
* [transaction setObject:object forKey:key inCollection:collection withMetadata:metadata]
* where the object is being inserted (value for collection/key does NOT exist at the moment this method is called).
**/
- (void)handleInsertObject:(id)object
forCollectionKey:(YapCollectionKey *)collectionKey
withMetadata:(id)metadata
rowid:(int64_t)rowid;
/**
* YapDatabaseReadWriteTransaction Hook, invoked post-op.
* Corresponds to [transaction setObject:object forKey:key inCollection:collection] &
* [transaction setObject:object forKey:key inCollection:collection withMetadata:metadata]
* where the object is being updated (value for collection/key DOES exist, and is being updated/changed).
**/
- (void)handleUpdateObject:(id)object
forCollectionKey:(YapCollectionKey *)collectionKey
withMetadata:(id)metadata
rowid:(int64_t)rowid;These 2 methods are defined for YapDatabaseExtensionTransaction, and you'll need to override them in your subclass. You'll notice that the hook methods pass you a bit of information that was discovered as the core layer was performing the operation. That is, whether or not this is an insert or an update. Your extension class may be able to take advantage of this knowledge, and optimize around it.
You'll also notice that the extension methods are passed a YapCollectionKey tuple object, and a rowid.
The YapCollectionKey class is a simple tuple class, and is used heavily within the YapDatabase internals. In particular, YapCollectionKey instances are used as the 'key' for the cache. So it ends up being much faster (and simpler) to pass around these immutable tuples, verses separate collection & key strings.
The rowid is the primary key of the row within sqlite. That is, if you inspect the 'database' table within the sqlite file, you'll see there is an INTEGER PRIMARY KEY rowid. There are a few implications to understand here (if you're not already familiar with sqlite):
- the rowid will never change (even if you vacuum the database)
- the rowid can be used to fetch all other information (including collection, key, object & metadata)
- its faster to fetch information from the database using the rowid, vs using just the collection/key
Basically, the collection/key is external facing information. But the rowid is what should be used internally. Thus, for developing extensions, you should likely be storing rowid's. Not only are they smaller (int64_t vs long string pairs), but they're faster.
Additionally, there are a LOT of heavily optimized methods that accept a rowid parameter. These were designed for internal use (which includes extensions). They're listed in YapDatabaseInternal.h. Here's a small sample. See the header for the full list.
- (YapCollectionKey *)collectionKeyForRowid:(int64_t)rowid;
- (BOOL)getCollectionKey:(YapCollectionKey **)collectionKeyPtr object:(id *)objectPtr forRowid:(int64_t)rowid;
- (BOOL)getCollectionKey:(YapCollectionKey **)collectionKeyPtr metadata:(id *)metadataPtr forRowid:(int64_t)rowid;
- (BOOL)getCollectionKey:(YapCollectionKey **)collectionKeyPtr
object:(id *)objectPtr
metadata:(id *)metadataPtr
forRowid:(int64_t)rowid;
- (BOOL)hasRowid:(int64_t)rowid;
- (BOOL)getObject:(id *)objectPtr
metadata:(id *)metadataPtr
forCollectionKey:(YapCollectionKey *)collectionKey
withRowid:(int64_t)rowid;
- (void)_enumerateKeysInCollection:(NSString *)collection
usingBlock:(void (^)(int64_t rowid, NSString *key, BOOL *stop))block;
// Many more methods like this...There is one last thing I want to say about rowids before moving on. Often times rowids are being used as a local cache for data coming from the server. And naturally, the collection/key values are coming from the server as well. This sometimes leads developers to think that they cannot use rowids, and instead they need to be storing the collection/key tuples in their extension. "But the rowids are different on each device. So yeah, I think I need to store the collection/key..." This is incorrect thinking. It does not matter what the actual value of the rowid is. Think of it, instead, as a "pointer" to the row that contains the collection/key (and object & metadata). So it doesn't matter if the rowid is 55 on your iPhone, and 66 on your iPad. Both point to the row with "foo/bar".
Rowids serve two very important purposes for extensions:
- They minimize data duplication. Rather than storing long collection/key string tuples throughout the entire database, we can instead store int64_t rowids. Which results in smaller database files.
- They help optimize sqlite operations. Because sqlite internally uses int64_t rowid values, we can make use of this with optimizations such as "INTEGER PRIMARY KEY".
Those using your completed extension will do something similar to this:
MyAwesomeYDBExtension *ext = [[MyAwesomeYDBExtension alloc] init...];
[database registerExtension:ext withName:@"awesome_sauce"];
The registerExtension:: method is going to:
- create a ReadWrite transaction
- ask your extension base class for an extensionConnection instance
- ask your extensionConnection for an extensionTransaction instance
- invoke 'createIfNeeded' on your extensionTransaction instance
This method is documented in YapDatabaseExtensionTransaction.m:
/**
* Subclasses MUST implement this method.
*
* This method is called during the registration process.
* Subclasses should perform any tasks needed in order to setup the extension for use by other connections.
*
* This includes creating any necessary tables,
* as well as possibly populating the tables by enumerating over the existing rows in the database.
*
* The method should check to see if it has already been created.
* That is, is this a re-registration from a subsequent app launch,
* or is this the first time the extension has been registered under this name?
*
* The recommended way of accomplishing this is via the yap2 table (which was designed for this purpose).
* There are various convenience methods that allow you store various settings about your extension in this table.
* See 'intValueForExtensionKey:' and other related methods.
*
* Note: This method is invoked on a special readWriteTransaction that is created internally
* within YapDatabase for the sole purpose of registering and unregistering extensions.
* So this method need not setup itself for regular use.
* It is designed only to do the prep work of creating the extension dependencies (such as tables)
* so that regular instances (possibly read-only) can operate normally.
*
* See YapDatabaseViewTransaction for a reference implementation.
*
* Return YES if completed successfully, or if already created.
* Return NO if some kind of error occurred.
**/
- (BOOL)createIfNeededThis is where you can create your sqlite table(s), or anything else you may need to do for your custom extension. A few things to note about the invocation of this method:
- it is invoked within a ReadWrite transaction
- the invocation of this method is the only thing that's going to happen in this ReadWrite transaction
- it is invoked every time the extension is registered (think every app launch)
You should be familiar with some of the standard extensions that ship with YapDatabase. Consider YapDatabaseView for a moment. It uses the concept of a "versionTag". If one changes the versionTag value between app launches, then the view automatically flushes its tables, and re-populates itself.
So where does it store the 'versionTag' value?
Storing configuration values (such as the versionTag) is common among almost every extension. So YapDatabase provides several useful utilities to simplify these tasks. The following are defined for YapDatabaseExtensionTransaction:
- (BOOL)getBoolValue:(BOOL *)valuePtr forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (BOOL)boolValueForExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (void)setBoolValue:(BOOL)value forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (BOOL)getIntValue:(int *)valuePtr forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (int)intValueForExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (void)setIntValue:(int)value forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (BOOL)getDoubleValue:(double *)valuePtr forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (double)doubleValueForExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (void)setDoubleValue:(double)value forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (NSString *)stringValueForExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (void)setStringValue:(NSString *)value forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (NSData *)dataValueForExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (void)setDataValue:(NSData *)value forExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;
- (void)removeValueForExtensionKey:(NSString *)key persistent:(BOOL)inDatabaseOrMemoryTable;(See YapDatabaseExtensionPrivate.h for declaration, YapDatabaseExtensionTransaction.m for implementation)
And thus, YapDatabaseView can check its previous 'versionTag' value with code like this:
NSString *previousVersionTag = [self stringValueForExtensionKey:@"versionTag" persistent:YES];YapDatabase maintains a dedicated table named "yap2" which can be used to store any persistent configuration information for your extension. It has the following structure:
| extension | key | blob |
The methods listed above allow you to specify the key and the blob (casted to whatever type you need). And the methods automatically use an extension value equal to the registeredName of your extension. In the example above, we registered an instance of our custom extension with the name @"awesome_sauce", so if we had code like this:
[self setStringValue:@"bar" forExtensionKey:@"foo" persistent:YES];Then we could inspect the "yap2" table and find this:
| "awesome_sauce" | "foo" | "bar" |
Keep in mind that you shouldn't need to access the "yap2" table directly. The utility methods handle it all for you. So storing and fetching configuration information for your extension is a simple one-liner.
If you're familiar with the general YapDatabase architecture, then the 3 extension classes should be familiar as well.