Operations vs Engineering

In many companies, you have the engineers and then you have the operations team. Engineers complain that the operations people aren’t nimble and agile. The operations people complain that the engineers never plan or keep them in the loop until the last minute. In many ways it’s an uphill battle for most organizations. Because I deeply care so much about infrastructure, it forces me to play a dual role. I have to understand what’s involved from the engineering effort, and what’s required to fulfill the needs of the operations team to keep things moving at light speed. One of the great things at Digg is that the two teams work closely together to achieve the same goals. Respect and guidance is key.

Under the hood

We’ve been making a lot of changes internally. Starting with providing more ownership to engineers, and investing in infrastructure that will scale and allow us the agility to make rapid decisions. A few months ago we moved the entire Digg codebase from Subversion over to Git. Ultimately we saw the headaches of merging in subversion and we had enough engineers already using it on open source projects, that we knew it was the right direction to improving our foundation as a company. Working at Yahoo, I quickly learned how valuable distributed computing, storage and logging can be for an engineering team to function. At Digg we have been been making investments into our data storage infrastructure (Hadoop & Cassandra), along with building out massive logging infrastructure that allows us the ability to collect metrics and reporting in real-time. The core infrastructure team at Digg has been using Hadoop, and our logging framework to mirror production traffic against our test clusters to analyze performance and do internal dark launches. Performance is critical to engineering and our cost of operations. We’ve been working hard on eventual consistency between multiple datacenters, multi-tier caching, front-end and CDN performance, automatic failover of services across datacenters, and building a new software stack that isn’t just scalable and fast, but is also cost effective and flexible.

Open Source

Digg’s support of open source has been truly refreshing; considering at other companies I’ve worked at it has proven difficult to get patches or new projects released. Not many companies realize the true value that open source provides. VP of Engineering, John Quinn, saw this early on. I was really happy to see someone on the executive management team actually understand its importance. It was one of the many reasons I joined Digg. In terms of open source at Digg, we are heavily involved in the community. We understand its importance to our infrastructure as well as how important it is to give back to the community. We have developers working on: Cassandra, Thrift, Scribe, Hadoop, Hive, and memcached to name a few. Along with a slew of our own projects over on Github.

Bringing it together

Without a solid foundation to really build upon, you’re left spinning your wheels on cruft, frustration and telling the product people, that new addition or change is going to take weeks to months, instead of days. I like thinking in days, and sometimes that surprises people. When your tools are automated and your foundation is solid, it really allows you to iterate, test and actually make a decision. We’ve all sat in meeting after meeting trying to come up with ideas and decisions of what to do, just to setup up another meeting. You have to have a balance of risk, foundation, culture, open source and most of all, talented people who can see the big picture.

We have many hard problems to solve here at Digg, and many more amazing ideas to build on. We are making remarkable strides in just the few months that I’ve started working at this company and I look forward to the road ahead…

Today, we’re also making some changes to reduce page payloads and minimize HTTP requests with subtle UI changes. By removing the 16px user icon from stories on the home page and other story lists, we’re reducing HTTP requests to Digg for a warm cache load by around 75%.

This is just the start of the performance initiatives that will unfold over the next few months and will result in order-of-magnitude speed-ups for many use cases.

Cheers,

John.

“If everything seems under control, you’re just not going fast enough.”, Mario Andretti

Since it was already necessary to abandon data normalization and consistency to make these approaches work, we felt comfortable looking at more exotic, non-relational data stores. After considering HBase, Hypertable, Cassandra, Tokyo Cabinet/Tyrant, Voldemort, and Dynomite, we settled on Cassandra.

Each system has its own strengths and weaknesses, but Cassandra has a good blend of everything. It offers column-oriented data storage, so you have a bit more structure than plain key/value stores. It operates in a distributed, highly available, peer-to-peer cluster. While it’s currently lacking some core features, it gets us closer to where we want to be than the other solutions.

We started thinking seriously about deploying Cassandra in production around three weeks ago. After looking at the site for something that would be a good fit, we settled on green badges. These badges appear on the Digg icon for a story when one of your friends has dugg it. This has been a problematic feature to support as we’ve grown, and corners had to be cut to keep it from breaking. They’re disabled on large pages (such as top in 365) and highly digg stories.

The fundamental problem is endemic to the relational database mindset, which places the burden of computation on reads rather than writes. This is completely wrong for large-scale web applications, where response time is critical. It’s made much worse by the serial nature of most applications. Each component of the page blocks on reads from the data store, as well as the completion of the operations that come before it.

Non-relational data stores reverse this model completely, because they don’t have the complex read operations of SQL. The model forces you to shift your computation to the writes, while reducing most reads to simple operations – the equivalent of SELECT * FROM `Table`.

CREATE TABLE `Diggs` (
  `id`      INT(11),
  `itemid`  INT(11),
  `userid`  INT(11),
  `digdate` DATETIME,
  PRIMARY KEY (`id`),
  KEY `user`  (`userid`),
  KEY `item`  (`itemid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
 
CREATE TABLE `Friends` (
  `id`           INT(10) AUTO_INCREMENT,
  `userid`       INT(10),
  `username`     VARCHAR(15),
  `friendid`     INT(10),
  `friendname`   VARCHAR(15),
  `mutual`       TINYINT(1),
  `date_created` DATETIME,
  PRIMARY KEY                (`id`),
  UNIQUE KEY `Friend_unique` (`userid`,`friendid`),
  KEY        `Friend_friend` (`friendid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

We’ve made a few changes to the way Digg links to external sites that may impact some folks in the SEO community. These changes reduce the incentive to post spammy content (or link spam) to Digg, while still flowing ’search engine juice’ freely to quality content. We’ve added rel=”nofollow” to any external link that we’re not sure we can vouch for. This includes all external links from comments, user profiles and story pages below a certain threshold of popularity.

This work was done in consultation with leading experts from the SEO/SEM and link spam fields, in an effort to lookout for the interests of content providers and the Digg community. As always, we will closely monitor these changes in the wild and iterate based on feedback.

Cheers,
John

Many in the community have expressed interest in understanding how our new search works.  We’re using Apache SOLR/Lucene which helped us scale horizontally and solved many of our relevancy issues as well as enabling discovery of new content through facets.  Beyond site search, the rich set of features has allowed us build a platform that enables other features such as Related By Source and Related By Keywords.  I’ve recently had the pleasure of doing a podcast with Grant Ingersoll from Lucid Imagination that discusses some of these details.   I hope this podcast gives some insight with regards to how we’ve built the new search platform.

‘Til next time,
Sammy

Just a quick note about our DUI JavaScript project: Two weeks ago I pushed v1.0.0 of DUI to Digg’s Github, and nothing appears to have melted, so as of right now I’m officially declaring it Ready For Prime Time(tm).

1.0.0 is a complete rewrite from the ground up. The API has changed significantly, and much cruft has been slain, so you should probably give the wiki a once-over before diving in.

Also, I’ll be speaking at jQuery Conference ‘09 in September. If you’re curious about DUI, feel free to attend my session and grill me about everything I did wrong ;)

Cheers,

- Micah

When I started with the company the entire engineering team would work on the same project together, everyone in the company would get together in a conference room for an afternoon or two and do QA, and when it was all done, we released it and moved on to the next project.  Fast-forward to today, we have multiple projects, all being developed in parallel, with QA testing each feature as the engineer writes it.  And as soon as it is finished and signed off on, it’s released.

In the beginning we used Subversion.  We checked out, branched, committed, merged, and tagged.  It was far from perfect, it didn’t make anyone happy, but it’s what we had.  And it worked, mostly.  It was simple to understand and pretty easy to use.  But it didn’t scale.  As our engineering team grew, more and more work was being done in the same code base, we started stepping on each others toes more and more.

The integration process (merging a feature into trunk for release) would sometime take days and result in numerous conflicts in the code.  It was up to the person doing the merging to resolve it himself or try and track down the original developer to determine which pieces needed to be kept.  This resulted in both a large number of bugs as well as code just simply disappearing.  Something that was costing us time, money, and impacting our users experience.  All of which we take very seriously.

We knew there must be a better way, so we started looking for it.  We tried upgrading from subversion 1.4 to 1.5, which has “merge tracking”.  We hoped it would make our integration process easier.  But after working with it for a few weeks we realized that the “merge tracking” feature was akin to putting lipstick on a pig.  And as nice as it sounded, it just wasn’t going to cut it.

Then everyone started talking about Git.  A few of us had heard about it, only a couple had ever used it, and mostly just for personal projects.  But it was the next cool thing.  It was GREAT!  It’s going to make our engineers more productive and make their lives so much easier.  It was like someone was handing us a nice tall cold glass of cherry kool-aid and telling us everything was going to be OK.  Would you drink it?  Well, we did.  Not the entire glass at first, just a little sip.  Then we realized, it wasn’t so bad, it actually felt kind of good.  So what did we do?  In true Digg fashion, we chugged the rest of the glass and asked for seconds.

So why is Git so great?  Well, simply put, it’s powerful and it’s distributed.

It allows you to do complex tasks with only a few commands.  It makes things like branching and merging almost completely thoughtless.  It even allows you, and goes so far as to encourage you, to rewrite history.  It really lets each engineer work the way they want to.  And when they’re ready, allows them to push their code upstream for others to see.  It manages code and changesets so drastically different from Subversion that it truly takes a complete mental shift about how you think about source control to even begin to understand it.  But believe me, it’s a good thing.

But Git does have it’s downsides.  At first it can be difficult to understand what is really happening behind the scenes when you run a command and if something does go wrong, you really need to understand how Git works internally to correct your mistake.  It also isn’t completely polished yet, there are no good GUI’s for it and the error messages you get when a command fails might as well be written in sanskrit.

Even with all of its downsides, its steep learning curve, confusing error messages, and lack of polished UI’s.  It works.  It works so well in fact that after switching to Git, we were able to reduce the time our integration process took from days to minutes.  Literally from 1-2 days down to about 30 minutes.  And, after it’s done, we have much more confidence that the merge was done correctly.  We’re also able to release code on a daily basis, with very minimal effort, something that was extremely painful to do under Subversion.

Stay tuned for part two.

Should Digg block IE6?

Currently, IE6 usage accounts for 10% of Digg visitors and 5% of page views on Digg. While this is down from 13% and 8% a year ago respectively, IE6 still accounts for a fairly large portion of Digg usage. That said, a lot of time is spent by Digg engineers supporting site activity like diggs, buries, and comments in IE6, and while it accounts for 5% of site traffic, IE6 accounts for only 1% of diggs, buries, and comments.

Based on the amount of activity and the relative rate of its decline, we’re likely to stop supporting IE6 for logged in activity like digging, burying, and commenting. Users of IE6 would still be able to view pages — just not logged in. This won’t happen tomorrow, but we’re thinking about doing it soon. If you have an opinion on this, hop into the comments.

Should Digg prompt IE6 users to upgrade?

A message suggesting the IE6 user upgrade seems like a logical approach. Then we wondered, “With a number of sites showing upgrade messages to IE6 users, why haven’t they already upgraded?” To find out, we ran a message to IE6 users on Digg asking, “Have 45 seconds? Help Digg by taking a quick three question survey.” The three questions were:

• What browser(s) do you use at work? (Please choose all that apply.) [IE, Firefox, Opera, Safari, Chrome, Other]
• What browser(s) do you use at home? (Please choose all that apply.) [IE, Firefox, Opera, Safari, Chrome, Other]
• You are currently using Internet Explorer 6. Why?

Based on the 1571 results, people use IE6 much more often at work.

This goes directly to why most folks use IE6: they don’t have a choice. Three out of four IE6 users on Digg said they can’t upgrade due to some technical or workplace reason.

Giving them a message saying, “Hey! Upgrade!” in this case is not only pointless; it’s sadistic.

We’re committed to developing to Web standards and building new ways to help you discover the best of the Web. Keeping an eye on what technologies folks use and why they’re being used is a big part of it.

Slainte,
Trammell

Today, we’ve introduced some exciting new additions to the Digg API, so that developers can create increasingly useful and interesting applications using Digg data.

One of the most requested features has been for a search endpoint and we’re excited to make it available in this release. This feature utilizes the same search functionality introduced a couple months back with the overhaul of Digg search and provides a powerful solution for finding specific content on Digg. You’ll be able to use the advanced shortcuts, common search tricks, as well as search by source (domain). See the Search API documentation for more details.

We’ve also added a series of related stories endpoints, so that you can find related information for any story on Digg. One variation finds stories similar diggers have dugg. Another returns stories with similar keywords. Finally, we’ve included an endpoint for favorites on Digg, which are an indication of the stories that people find exceptional in some manner.

These features are just the beginning of some changes that we’re currently working on and plan to introduce in the coming months. These will include endpoints for participating — such as Digging and burying — so that even richer and full featured applications can be created off the Digg platform.

Finally, a significant but subtle change we want to announce is a reworking of our API license. We want to give developers more control over the applications they develop by removing most of the commercial limitations. Developers should now have the confidence that they can benefit from the works they create using the Digg API, with full ownership free of fees!

We hope you enjoy these changes and, as always, welcome your feedback.

Cheers,

John

Although we usually wait until a feature is fully baked before showing off the technology behind it, we’re just too excited about this new project to keep it to ourselves. Digg’s Front End Architecture team has been experimenting a lot with performance improvements, and we’re just about done with one that we think will change the way you get data from high-traffic websites. First though, a bit of background:

One of the ways that high-performance websites like Yahoo suggest speeding up load times is by reducing the number of HTTP requests per page. We started thinking about what we could do to reduce HTTP overhead, and where we could get the biggest benefits from it. Well, one thing led to another and the next thing we knew we were talking about writing a generalized framework for bundling files, sending them through a single request, then separating them for use once they head down the pipe.

We call this technique MXHR (short for Multipart XMLHttpRequests), and we wrote an addition to our Digg User Interface library called DUI.Stream to implement it. Specifically, DUI.Stream opens and reads multipart HTTP responses piece-by-piece through an XHR, passing each chunk to a JavaScript handler as it loads.

Why do this? Well, DUI.Stream will allow developers to drastically improve the speed of uncached page loads by bundling most of their resources into a single HTTP request, with a single time-to-first-byte and no request throttling by the user agent. Additionally, the size of the response has no effect on the rendering time of each chunk, as the client handles each piece of the response on the fly and can inject it into the DOM for rendering immediately, in the exact order you specify. On a high traffic, high-activity site like Digg, we have to display incredible amounts of data on each permalink — typically hundreds of user images within the first 50 comment threads on a page alone, not to mention the UI chrome and actual comment data. (You can see this for yourself: notice the number of HTTP requests that queue up when you expand a page of comments). So our primary use case for DUI.Stream is turning that first long, arduous page load on an empty cache into something nearly indistinguishable from a page of data with fully cached resources.

Let’s talk a bit about the architectural benefits of implementing MXHRs with DUI.Stream. Back when the web was based largely on a page metaphor (i.e.: one central document with external references), whenever you loaded the page, the page requested its images, stylesheets, etc, then you were done. These days you’re just as often loading an application; the page progressively enhances into a stateful UI by loading extra stylesheets, scripts and a whole mess of UI chrome after the initial request. Yet, we’re still using the old model flow of get markup –> render markup –> request external resources –> load and display externals.

Take our modal login dialog box for example. In order to reduce requests we bundle its JavaScript in with the rest of the page, we put its CSS up in the header with the rest of the styles, then we request only the markup for the dialog box, render it, and let it fire its own HTTP requests for the images that make up its chrome. In this broken model, HTTP connections and rendering behaviors split our UI architecture up into different parts of the page that all render at different times at the browser’s discretion. Even if we put everything into one cohesive structure and loaded the CSS link, script tag and markup together, they’d still all fire their own HTTP requests and the images would still come in afterwards on the first page load. This just won’t do.

Now, let’s rethink how our login dialog could work using DUI.Stream. We can request a Stream that contains everything needed to render and use the dialog box. As each part comes in, it gets passed through to be built, and renders immediately with no image backfill or delayed JS behavior. The DUI.Stream framework can then pass those resources back into cacheable elements for our next page load, which can happily 302 its way quickly through the rendering process. Pretty sweet right? Right.

Before we get into actual code, I have to stress that the DUI.Stream client, with all its aforementioned badassitude, is an alpha release — it’s better than a proof of concept, but it’s definitely not ready for prime time. Think of it like Miles and Hurley’s time travel discussion on Lost:  There are still a few plot holes, but what we’ve seen so far is pretty cool. With that in mind, let’s save the talk about future features for a bit later and get into what DUI.Stream does right now.

Here’s a TLDR example:

var s = new DUI.Stream();
 
s.listen('text/html', function(payload) {
    $('#stream').append(payload);
});
 
s.listen('complete', function() {
    alert("D. Plainview: I'm finished!");
});
 
s.load('testStreamData.php');

DUI.Stream works on a pretty simple mechanism: create a DUI.Stream, register listeners for each mimetype you plan to send (as well as an optional onComplete), then send a request to a URL that implements an MXHR responder. I’ll leave the details of DUI.Stream’s inner-workings to those of you who want to look through the source. As always, our code is released under a open-source license so if you find any fugly parts, feel free to let us know or send us a patch.

You can see the demo code in action, and I’ve included some rough timing numbers to demonstrate just how much faster DUI.Stream is than a regular uncached page load. YMMV with the demo, so don’t be surprised if it falls over in your browser or occasionally reports wacky numbers. Check out the roadmap below for cross-browser info. Also, we have a demo that handles images that performs atrociously well in Firefox and Safari, but at present IE doesn’t play nice with it (shocking, I know). You can test drive it at your own risk.

If you’re curious about responders, we’ve thrown together a few demos in Python, Ruby, Perl and Java in the DUI.Stream GitHub repository. We’ll be releasing a full set of documentation once we hammer out the rest of the DUI.Stream client. Rest assured you won’t have to implement anything based on a blog post and some demos.

Here are the things still on our short-term roadmap:

• Cache detection. If you wanted to implement DUI.Stream right now, you’d have to switch between regular requests and MXHRs on your own. We don’t recommend that.
• Background caching. A big part of implementing this is going to be loading the MXHR through DUI.Stream, then turning any cacheable chunks into their normal, cache-friendly tags.
• Support for multiple headers per chunk. Specifically, we’ll be adding a set of custom headers to allow for UI-specific information to be sent, like CSS selectors for greater control in handlers.
• XMLHttpRequest.multipart support. Right now we’re not using this flag, since we aren’t keeping connections open for a server-push implementation yet. Like I said, It’s an alpha ;)
• IE7 and IE8 object tag workarounds. In the current build, bouncing binary streams into object tags in IE has some interesting results.

Anyways, I hope you like what you’ve seen so far. Several of my teammates have been busting their asses to get this ready to show off — specifically Jordan Alperin, Arsenio Santos, Chris Goffinet, and Arin Sarkissian — and we’re all pretty excited about it. As always, if you have any questions, comments or thoughts, let us know!

Cheers,
- Micah