I’ve been hinting at a mysterious piece of infrastructure, called IDDB, for some time in various talks, blog posts and mailing list posts. Today, with the introduction of Digg’s DiggBar and URL minifying service, IDDB is finally powering a significant feature. I say that because it’s actually been quietly powering user IM’s and links for about a month with no issues (this is why that feature temporarily disappeared a few weeks ago only to reappear after a major data migration).

So what, exactly, is IDDB? IDDB is a way to partition both indexes (e.g. integer sequences and unique character indexes) and actual tables across multiple storage servers (MySQL and MemcacheDB are currently supported with more to follow). It started out as a harebrained idea that a few guys from operations and I would chat about during lunch. After a few lunches Ron and I decided to put the marker to the whiteboard and hammer out the details. A few afternoons later we had the basics whiteboarded and I’d prototyped a working package.

From there Ian and the Core Infrastructure team took over implementing and refining IDDB. There were tools to be written for operations, features to be fleshed out and tests to be written (In fact, 41% of IDDB’s code is its unit test suite). In the end some of the features that were cooked up for IDDB include:

• Ability to partition data across multiple storage nodes using arbitrary node assignment. The hashing framework allows node assignment to be pluggable (e.g. We could create hashing algorithms that hash ID’s by colocation facility, geological location of the user, racks, etc.). In addition to this, we allow for multiple types of storage nodes.
• Ability to break IDDB indexes, which store ID and location meta data, across up to 16 machines.
• Each ID maps to N storage nodes and has an individual status on each node. All data, by default, is written to three storage nodes. MySQL and MemcacheDB both support replication so their slaves are added to each ID as a read-only node.
• Each ID type can be served from its own cluster of servers. This means that the user ID sequence can be on a totally different cluster than the user email unique character index.
• IDDB utilizes Gearman to move data around. For instance, we can take a user who’s consuming a lot of resources and migrate them to a less loaded set of storage nodes. We also use this to find ID’s that are in a state of error to fix them (e.g. A user ID requires 2 copies, but there’s only 1 so we lock it and make another copy).
• Storage nodes can be arbitrarily added to the pool at any time and new ID’s will instantly start mapping to them. Additionally, we can set a storage node’s status to “full”, which means it can no longer accept new ID’s, but will continue to accept new data for existing ID’s. When combined with our migration tools we can elastically grow the storage clusters out and rebalance ID’s across them as needed. This also means we can keep heterogeneous hardware operating in unison (e.g. Older machines can hold 10,000 users, but new ones can hold 25,000).
• All meta data about ID’s, which is pretty much entirely static, is kept in Memcached to reduce reads from the index cluster.
• Ability to track number of queries being ran against a storage node or against a single ID.
• An entire suite of CLI utilities to manage the clusters.

What’s nice is that IDDB abstracts all of this, making it quite easy for developers to partition data without having to worry about the basics. Here’s how you’d fetch a user record and query for the user’s IM links:

<?php

require_once 'IDDB/ID.php';

// Fetch a user's IDDB record, which contains basic meta data and location data.
$id = IDDB_ID::factory('User', 1234);

// Get all of the user's data from a random node.
$links = $id->db()->getAll('SELECT * FROM UserLinks WHERE userid = ?', array($id->id));
print_r($links);

?>

Pretty simple. But what about creating a new ID? Let’s say we’re creating a new user and want to add some IM’s to their table. What would that look like?

<?php

require_once 'IDDB/ID.php';

// Creates a new unique, auto-increment, ID in IDDB.
$id = IDDB_ID::create('User'); 

// The execute() method will write this query to all of the ID's writable nodes within individual
// transactions. If it fails to write on all nodes then the exception is surfaced. If it fails on 1
// of, say, 3 writes it will mark the ID as being in an error state on that 1 node while being
// live on the other two nodes.
$id->db()->execute('INSERT INTO UserIMs SET userid = ?, service = ?, handle = ?', array(
    $id->id, $_POST['service'], $_POST['handle']
);

?>

The DiggBar and URL minifying service is powered by a 16 machine IDDB cluster, which includes 8 write masters in the index and 8 MySQL storage nodes. It’s, to date, the largest IDDB cluster Digg has pushed into production, but we have plans for much bigger IDDB clusters.

On a final note I really want to call out IDDB’s test suite and how putting extra effort into tests has helped us rapidly iterate over IDDB. The test suite contains 2,300 lines of code, 990 tests and a whopping 8,081 assertions! In fact, IDDB’s test suite has surfaced (and fixed!) a number of bugs in PHPUnit itself. It’s imperative that such a crucial piece of code be completely unit tested and I’m insanely impressed with Ian’s work in this area.

Thanks!

Joe