Read at Joe’s

One of the things that drives everyone nuts about Twitter is its unreliability. In fact, it’s having a little case of the no-updates this morning (2008.04.20). This is a less frequently discussed but ultimately, I think, more important disadvantage to walled gardens. If you rely on a service and it goes down for maintenance or failure, or it is subjected to any one of several denial-of-service attacks by hackers, you’re out of luck. You’re subject to how much, or more properly, how little time and money the service’s administrators or financial backers have put into site dependability.

So, what could be done about this problem? The answer would be to design a service that is distributed instead of centralized. A distributed Twitter service would operate like email. There’s no single point of failure for email because there’s no single super email server or service through which everything flows. Email is based on a protocol, not a single service, or even group of services. Anyone can run an email server and exchange mail with anyone else running a server. I think sometimes people would like email to go down for a few days because they feel overwhelmed by it, but that’s a different blog post entirely.

Of course, you might say it all flows through the Internet, but the net itself is distributed. Yes, there are backbone subnets that carry huge amounts of traffic and would degrade your service if they went down (and they sometimes do), but there are well-known ways to re-route traffic around failed links.

Twitter could most definitely be distributed. Over the past few months there have been some real developments toward this end, and conversation in the development community about ways to do it. The single most complete development done so far is Prologue, a WordPress theme that allows several people to “co-author” blog posts and share their “update stream” via RSS. It’s been most talked about as a Twitter for corporate co-workers that in a certain sense gives the long-desired groups capability to Twitter.

Prologue is a step in the right direction and hints at what is possible, but is not a general solution that would scale to accommodate thousands of users. The guys at WordPress that developed Prologue have said they’re not particularly interested in developing a fully distributed Twitter-clone, but in the hopes that someone else might pick up the ball and advance it, they’ve made the code open and available under an open-source license.

I’ve been thinking a lot about this problem myself and in this introductory post I’d like to outline what I think are the major requirements and architectural issues to be solved in creating a fully distributed microblogging platform. Do you have any thoughts to offer on this? Please comment!

1. Open
   Although it’s certainly possible to design a closed microblogging architecture (ie, distributed but a proprietary implementation), it’s certainly in the general user’s interest to make it open, which would allow for competing implementations. Why break down a walled garden only to create another one?
2. Protocol -defined
   In order to be open and distributed, the architecture must defined by a protocol, not by a server program or database schema, or other implementation artifact.
3. Minimal
   Occam’s Razor should be the guideline. A new microblogging architecture should specify as little as possible in terms of UI, security techniques, etc. OpenID is an excellent example of a standard that leaves many important features (such as authentication technique) to the various implementations.
4. Privacy-capable
   I think there is a rising concern and general wariness of the completely open and public nature of many social media and social networking applications. Users want to be sure they know who sees their posts and control who can see what “friends” they have. So, there should be a general capability to control access to your microblog and its meta-data.
5. Extensible, Forward-Compatible
   It should be possible for an implementation to add new capabilities without rendering previous versions inoperative or non-interoperable.
6. Scalable
   The architecture should be scalable to the entire internet. A given user should be able to subscribe to or service subscriptions from tens of thousands of other users.
7. Efficient
   In order to fulfill #6, the architecture must be efficient of network and computing resources. In particular, this would argue against polling RSS feeds as a way to collect input. Polling a blog once a day works, but a microblog stream like Twitter needs to react almost immediately.
8. Standards-based
   Whenever possible, new open architectures should utilize existing standardized protocols and data formats whenever possible. This does not mean that a new protocol or format is out of the question, but there had better be a very good reason why an existing standard cannot be used.
9. Open-source
   While proprietary implementations of a standard cannot be prevented, open source implementations should be favored, especially with respect to security issues. Only by inspecting the source can one be assured there are no easter-egg back doors or other weaknesses.

Tags: distributed, microblogging, twitter