A short while back, an email showed up in my inbox, sent to an old and long-forgotten address. The subject line was "[email protected] Newsletter." That was a project originally launched ten years ago, and the administrators wanted to commemorate the anniversary with a thank you note to those who had been part of it -- a nostalgic nod to a pretty awesome bit of technology and a milestone in computing history.
[email protected] was meant to search for anomalies in space, like pulsars and other exotic astronomical phenomena. It was a project that required vast computing power -- in this case, to scour a myriad of images in the sky in a range of wavelengths and to identify slight variations in star brightness over time. Aside from the noble goals of the project, what was really exciting was that it was built with a distributed algorithm, using volunteers who donated their computer idle time to carry out complex computing. Anyone with an Internet connection could download a program that would run in the background and communicate with a server to get data and instructions, process tasks and report the results back.
Ten years on, there are more than 300,000 volunteers running [email protected] from over 220 countries, representing a whopping 1,000 petaflops of computing power -- that would rank in the top 20 list of the world's most powerful supercomputers.
The project has generated numerous papers and was instrumental in the discovery of a number of radio-wave pulsars and other curious space objects. [email protected] is now also analyzing Gravitational Wave data. But the scientific value of the project is not what struck me as I thought back over the ten years of donating my CPU to science. What came to mind, rather, was the pivotal part that this technology played in the history of computing and the far-reaching social and economic disruption it facilitated.
Distributed computing has been around since the 1970s. As the Internet evolved, by the late 1990s, it was applied to connected, always-on, personal computers. One of the first projects that used this wide-scale communication and sharing technology was [email protected], a project out of Berkeley that analyzed the data from huge arrays of radio telescopes to search for signs of extraterrestrial life. Advances in distributed processing and shared computing led to improvements in two closely related, subsidiary technologies: Peer-to-Peer Computing and Distributed File Sharing.
When Napster burst onto the online scene in 1999, it was one of the first commercial applications to take advantage of those technologies. Though it only lasted two years, Napster and the music sharing culture that it spawned set in motion incredibly disruptive changes in the way people consumed entertainment, and put us on a path that ultimately led to things like BitTorrent, Hulu and Netflix.
Napster was eventually shut down as a result of legal action taken against it for copyright infringement by the recording industry. But even as Napster pulled the plug on its servers, another company stood poised to take its place at the vanguard of the file-sharing universe. KaZaA, out of Estonia, relied more heavily on Peer-to-Peer and de-centralization of content and didn't face the same challenges as Napster. That program was wildly successful with more than 4 million active users and an install base numbering orders of magnitude above that.
On the auspicious day of August 29, 2003, every KaZaA user woke to find that another program had automatically downloaded and installed itself. The program allowed communication using a technology called Voice Over Internet Protocol (VoIP). It was based on essentially the same Peer-to-Peer discovery mechanism as KaZaA, and purportedly allowed any user to talk to any other user, anywhere in the world, for free. And the rest, as they say, is history.
By piggy-backing off of KaZaA's existing user base, VoIP was able to overcome the huge barrier of the Network Effect -- a term from economics that describes a service whose value is dependent on the number of subscribers using it. From day one, the install base was large enough for people to see the benefit in using it, and though the technology had been around for almost a decade, no commercial program had garnered any substantial success. Competitors not only had to contend with the ubiquitous distribution of KaZaA software, but also with the radical business model offering communication for free.
The avant-garde programmers at Berkeley, the intrepid researchers looking for radio signals from ET, the astronomers, biologists and chemists who used and improved distributed computing through volunteer CPU donors to carry out their work, could never have known that their efforts would eventually lead to extraordinary changes in society, economics and even geo-politics.
Peer-to-peer, file sharing, VoIP, social networking -- these technologies have led to a world of connected communication, on-demand entertainment, and content and data sharing and collaboration. As Bilbo once told Frodo of the Shire, "It's a dangerous business, going out your door. You step onto the road, and if you don't keep your feet, there's no knowing where you might be swept off to."
Happy Tenth Birthday, [email protected]
— Baruch Sterman, Vice President of Technology Research, Vonage