Software, Dragonflies and Reciprocal Altruism

gistfile1.txt

Samuel J. Greear
15 March 2010


    Many of the best ideas and implementations of those ideas in the realm of
computing come about when a single individual decides to sit down and create
something. It may be a new feature or entire system, or perhaps a new take on an
old idea. Perhaps popularized by Eric S. Raymond, in software engineering
parlance this is sometimes referred to as “scratching an itch” (Raymond, pars.
6-8). DragonFly BSD is a liberally licensed Unix-like operating system
compatible with commodity x86-derived personal computer and server hardware.
Numerous software developers have found common ground with DragonFly BSD, lured
by the operating systems history, rationale, goals, features and future
prospects. These developers contribute their energies to DragonFly BSD on an
ongoing basis with the only tangible rewards coming in the form of personal
satisfaction.

    DragonFly BSD comes from relatively humble beginnings. Bell Labs own history
of Unix tells that it was conceived by a researched named Ken Thompson, and that
it was a project originally intended to scratch some personal itches (In the
beginning). According to Bell Labs the defining time in the life of Unix was
1976-77. It was during this time that Ken Thompson taught a course on the
subject as a visiting professor at the University of California-Berkeley (UCB).
After being shown the light of day at UCB the system began to garner a great
deal of interest. Couple this with the fact that Berkeley's own Unix derivative,
Berkeley Unix, hosted one of the very first TCP/IP stacks, which serve as the
foundation of the Internet, and it was a recipe for explosive growth (Sharing).

    The next step on the path to understanding the history of DragonFly BSD lies
through the history of its parent operating system, which was based on the
Berkeley Software Distribution or BSD. According to the FreeBSD project, it
began in 1993 as an outgrowth of the work originally done at Berkeley. The
FreeBSD project spent seven years turning the original BSD code into a
world-class operating system. Between the years 2000 and 2003 the project's
focus turned toward “advanced multiprocessor and application thread support“
(About). According to the FreeBSD 5.0 Release Announcement, the means to
achieving this goal, which is primarily to satisfy performance demands on modern
hardware, was to add fine-grained synchronization to the operating system kernel
(Long).

    DragonFly BSD was forked, or derived from, FreeBSD in June of 2003. It was
forked from an earlier version than the current FreeBSD release to avoid
carrying with it the many warts that had been imparted on the most recent
version FreeBSD in the name of multiprocessor performance (Dillon).

    There are many itches that warrant scratching. DragonFly BSD was originally
forked by Matt Dillon, a long-time FreeBSD contributor, in order for him to
pursue what he perceived as a better path than the one FreeBSD had taken. This
path included not only the methods by which multiprocessor performance would be
achieved, but many other features as well. The biggest among these is single
system image clustering. Where a heterogeneous array of systems could be used
together as if they were a single computer with minimal extra effort or
configuration. The fundamental changes required to support clustering would have
been too disruptive to be allowed to take place in FreeBSD (DragonFly BSD:
history).

    You have an itchy hand, but my arm itches. That is to say, oftentimes when
an individual begins an interesting or motivating project they attract
collaborators. In some cases these collaborators are interested in roughly the
same goals as the originator. However, in most cases the end result is not as
important as the means to that result. No matter what your desired end result is
in a software engineering project, if your plans are ambitious enough you will
find a great deal of overlap with other projects. Many of the regular
contributors to DragonFly BSD are not expressly interested in the long-term
clustering goals of the operating system. Rather, they have found facets of the
operating system developed in support of those long-term goals that coincide
with their own personal itches. They then contribute to the projects long-term
goals by scratching at those personal itches in the form of source code,
documentation, commentary or any of a number of other means. An open,
collaboration-friendly and ambitious software engineering project, then, has
relatively few fixed goals but an ever-changing number of them. The trick is
keeping the whole thing moving in the right direction.

    Scratching a computer or software engineering itch isn't always about
creating something new. It is usually about doing something better, faster or
even just more elegantly. To this end DragonFly BSD has implemented a number of
concepts first seen in other places. One feature of DragonFly BSD similar to the
virtualization features of many operating systems before it but with a bit of a
twist is the virtual kernel. A virtual kernel is an actual operating system
kernel, the core of the operating system, compiled to run in user space. For
developers, this means that a great deal of previously cumbersome development
can now be done without rebooting the entire system. For users, it can be used
as a means of abstraction or resource isolation (Wildner).

    DragonFly BSD has introduced a number of interesting new features to the
world of operating systems. One of these novel features is called a serializing
token. Probably the most common serialization primitive, or method by which a
shared object may be safely accessed by multiple processors within a system, is
known as a blocking mutex. Blocking mutexes ensure safety through absolute
correctness in accessing the shared objects, which can be hard to satisfy,
especially when adding support for multiple processors to a legacy system. This
can easily lead to catastrophic phenomenons known as deadlock or priority
inversion. With serializing tokens, access to a piece of shared data is only
blocked while the thread holding the token is active. The trade-off is that
software performing synchronization with serializing tokens must actively detect
race conditions. However, serializing tokens are immune to deadlocks and a great
deal of code required to ensure correct mutex operation in this regard can be
avoided (Serializing tokens).

    The most recent and one of the most interesting features is the HAMMER
filesystem. HAMMER was created in direct support of the long-term clustering
goal. There are a number of features that set HAMMER apart from other file
systems. It was designed specifically for very large modern disks, many other
file systems do not handle modern disks optimally. In most file systems, when a
file is overwritten it is not actually overwritten internally. Both old and new
versions will continue to physically exist on the disk until the file system
needs the space used by the old file. In software engineering parlance, the old
file is orphaned. In HAMMER, this artifact of optimal file system operation is
turned into a feature. It keeps track of each old version and allows you to view
the history of a file. HAMMER also supports mirroring, or the near-real time
cloning of one live and active file system to another in another computer system
(DragonFly BSD: hammer).

    It is possible that the single system image clustering goals may never be
achieved. Others have tried and failed. Even if this unfortunate eventuality
were to materialize, DragonFly BSD could not be considered uninteresting or a
failure. It has already contributed great ideas and code under extremely liberal
terms to the global software ecosystem. This software is sent out into the world
with many hopes and dreams attached, but primary among these is that those who
it inspires to scratch their own itch will be altruistic in return. It has
already motivated and inspired individuals who are now scratching their itches
using DragonFly BSD as a sandbox, or elsewhere, but undoubtedly took a selection
of ointments with them. The future is certain to be unknown, but if the past is
any indicator the future for DragonFly BSD will be colorful and interesting.


Raymond, Eric S. "The Mail Must Get Through". The Cathedral and the Bazaar. 15 Mar. 2010.
    <http://catb.org/esr/writings/homesteading/cathedral-bazaar/ar01s02.html>. 

"In the beginning: botched acronyms". The Creation of the UNIX* Operating System. Bell Labs.
    15 Mar. 2010. <http://www.bell-labs.com/history/unix/acronyms.html>. 

"Sharing UNIX with the rest of the world". The Creation of the UNIX* Operating System.
    Bell Labs. 15 Mar. 2010. <http://www.bell-labs.com/history/unix/sharing.html>. 

“About the FreeBSD Project”. FreeBSD Handbook. 15 Mar. 2010.
    <http://www.freebsd.org/doc/handbook/history.html>.

Long, Scott. "FreeBSD 5.0-RELEASE Announcement". 15 Mar. 2010.
    <http://www.freebsd.org/releases/5.0R/announce.html>. 

Dillon, Matthew. "Announcing DragonFly BSD!". 15 Mar. 2010.
    <http://lists.freebsd.org/pipermail/freebsd-current/2003-July/006889.html>. 

"DragonFly BSD: history". 15 Mar. 2010. <http://www.dragonflybsd.org/history/>.
 
Wildner, Sascha. "DragonFly On-Line Manual Pages". 15 Mar. 2010.
    <http://leaf.dragonflybsd.org/cgi/web-man?command=vkernel&section=7>. 

"Serializing tokens”. Wikipedia. Wikimedia Foundation, Inc. 15 Mar. 2010.
    <http://en.wikipedia.org/wiki/Serializing_tokens>. 

“DragonFly BSD: hammer”. 15 Mar. 2010. <http://www.dragonflybsd.org/hammer/>.