See other formats

Full text of "Open Sources 2.0 : the continuing evolution"

ted by Chris DiBona, Danese 
Mark Stone With a Foreword by Kim Polese 




THE CONTINUING 

EVOLUTION 



O REILLY 



open sources 2.0 



XX 





open sources 2.0 

Edited by Chris DiBona Danese Cooper Mark Stone 



IHE CONTINUING 
EVOLUTION 

XX 
XX 



O REILLY* 

Beijing Cambridge Koln London Paris Sebastopol Taipei Tokyo 



Open Sources 2.0: The Continuing Evolution 

Edited by Chris DiBona, Danese Cooper, and Mark Stone 

Copyright 2006 Chris DiBona, Danese Cooper, and Mark Stone. All rights reserved. 
Printed in the United States of America. 

Published by O Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. 

O Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also 
available for most titles (safari.oreilly.com). For more information, contact our corporate/institutional sales 
department: (800) 998-9938 or corporate@oreilly.com. 

Executive Editor: Mike Hendrickson 

Production Editor: Jamie Peppard 

Cover Designer: Mike Kohnke 

Interior Designer: Mike Kohnke 

Printing History: 

October 2005: First Edition. 



The O Reilly logo is a registered trademark of O Reilly Media, Inc. Open Sources 2.0 and related trade 
dress are trademarks of O Reilly Media, Inc. 

The essays in Open Sources 2.0are licensed under the Creative Commons Attribution-NonCommercial- 
NoDerivs 2.5 license. To view a copy of the license, send a letter to Creative Commons, 543 Howard 
Street, Fifth Floor, San Francisco, CA 94105, visit http://creativecommons.Org/licenses/by-nc-nd/2.5/, 
or see Appendix B. 

Many of the designations used by manufacturers and sellers to distinguish their products are claimed 
as trademarks. Where those designations appear in this book, and O Reilly Media, Inc. was aware of a 
trademark claim, the designations have been printed in caps or initial caps. 

While every precaution has been taken in the preparation of this book, the publisher, editors, and authors 
assume no responsibility for errors or omissions, orfor damages resulting from the use of the information 
contained herein. 



^ This book uses RepKover? a durable and flexible lay-flat binding. 



ISBN: 0-596-00802-3 



XX 








Table of Contents 



Foreword: Source Is Everything ix 

Kim Polese 

Acknowledgments xiii 

List of Contributors xv 

Introduction xxv 

Chris DiBona, Danese Cooper, and Mark Stone 

SECTION 1 . Open Source: Competition and Evolution 1 

1. The Mozilla Project: Past and Future 3 

Mitchell Baker 

2. Open Source and Proprietary Software Development 21 

Chris DiBona 

3. A Tale of Two Standards 37 

Jeremy Allison 

4. Open Source and Security 57 

Ben Laurie 

5. Dual Licensing 71 

Michael Olson 

6. Open Source and the Commoditization of Software 91 

Ian Murdock 



7. Open Source and the Commodity Urge: Disruptive Models 

for a Disruptive Development Process 103 

Matthew N. Asay 

8. Under the Hood: Open Source and Open Standards Business Models 

in Context 121 

Stephen R. Walli 

9. Open Source and the Small Entrepreneur 137 

Russ Nelson 

10. Why Open Source Needs Copyright Politics 149 

Wendy Seltzer 

11. Libre Software in Europe 161 

Jesus M. Gonzalez-Barahona 
Gregorio Robles 

12. OSS in India 189 

Alolita Sharma and 
Robert Adkins 

13. When China Dances with OSS 197 

Boon-Lock Yeo, Louisa Liu, and Sunil Saxena 

14. How Much Freedom Do You Want? 211 

Bruno Souza 

SECTION 2 . Beyond Open Source: Collaboration and Community 229 

15. Making a New World 231 

Doc Searls 

16. The Open Source Paradigm Shift 253 

Tim O Reilly 

17. Extending Open Source Principles Beyond Software Development. . . 273 

Pamela Jones 

18. Open Source Biology 281 

Andrew Hessel 

19. Everything Is Known 297 

ugene Kim 

20. The Early History of Nupedia and Wikipedia: A Memoir 307 

Larry Sanger 

21. Open Beyond Software 339 

SonaliK. Shah 

22. Patterns of Governance in Open Source 361 

Steven Weber 

23. Communicating Many to Many 373 

Jeff Bates and Mark Stone 



vi xx Table of Contents 



SECTION 3 . Appendixes 397 

A. The Open Source Definition 399 

B. Referenced Open Source Licenses 401 

C. Columns from Slashdot 417 

Index. . . 423 



Table of Contents ** vn 



XX 
XX 



Foreword: Source Is Everything 



The software industry has always been caught between two perspectives: one 
anchored in supply, the other in demand. To the market s supply side (the vendors), 
commodities and "commoditization" have always been threats. To the demand side 
(the customers), commodities have always been useful. 

The latter view is winning, thanks to open source. And we re only beginning to dis 
cover how much larger the market will be, now that it s filling with useful open 
source commodities. These commodities, in most cases, have little or no sale value, 
but are useful for building countless other businesses. The combined revenues of 
those businesses will far exceed revenues of companies that make their money sell 
ing software. 

Use value precedes sale value in every market category. Think about it. Agriculture 
started with gardening. Textiles started with weaving and knitting. Meat packing 
started with herding. Construction started with hut building. 

What did the software industry start with? In a word, programming. As Eric S. Raymond 
said in the first chapter of the original Open Sources, "In the beginning were the Real Pro 
grammers." To Raymond, real programming was both legacy and destiny a source that 
began with "guys in polyester shirts, writing in machine and assembler and FORTRAN," 
and ran through Unix programming and the free software movement, to arrive at "Linux 
development and mainstreaming of the Internet." 



That latter phrase captured where the industry was when Open Sources was pub 
lished in January 1999. The Internet, Raymond noted, "has even brought hacker cul 
ture to the beginnings of mainstream respectability and political clout." 

A half-decade later, open source has grown far beyond the mainstream. It has 
become the bedrock over which the mainstream flows. Today it is hard to find a For 
tune 500 company with an IT infrastructure that does not depend, in some funda 
mental way, on open source software. 

The Internet mainstreamed programming by putting every programmer zero dis 
tance from every other programmer in the world. Supported by this extreme conve 
nience the demand side began supplying itself in a global way. "Real Programmers" 
were back in power. This time, however, real programmers were a legion, not a mere 
handful, and the tools they needed could be found on any PC, not just on the infra 
structure inside large organizations. 

Today power comes from everybody who creates anything that s useful to anybody 
and that any other programmer can improve. 

Today there are hundreds of thousands of hackers, perhaps millions. Whatever the 
number, many more will soon arrive from Asia, South America, Africa, and other for 
merly Net-less places all over the world. 

The technology sector s industrial age the one in which manufacturers built plat 
forms for silos in which customers and users were trapped will, in retrospect, 
appear to be an early growing stage: a necessary but temporary step toward a healthy 
and mature marketplace. 

Let s give credit where it s due: developing software and hardware in the early days of 
the computing industry was like settling Mars. Every company had to build its own air 
tight habitat, from the ground up. Making hardware and compatible software inside 
your own habitat was hard enough. Trying to become interoperable with anybody 
else s environment was nearly unthinkable. Even within large companies like IBM, 
whole systems were incompatible with whole other systems. Remember Systems 34, 
36, and 38, which used twin-ax cabling while IBM mainframes used co-ax? 

All these closed habitats naturally fell in to fighting. The press supported the battling- 
vendor view of the marketplace, which became a form of entertainment, dramatized 
via continuous coverage of the vendor wars. 

The Net obviates the need to build closed habitats. You don t need to make your own 
bedrock anymore. Open source commodities provide all the base infrastructural 
building materials you need, and then some. 

Naturally, the old supply side felt threatened by that. For a while. 



Foreword: Source Is Everything 



Then the demand side (the programmers) inside those silos began using open source 
to build solutions to all kinds of problems. Today IBM, Novell, HP, Sun nearly 
every big platform and silo company other than Microsoft have shifted their prod 
uct strategies to take advantage of abundant open source commodities in the market 
place. They also contribute, in most cases, to the development projects that continue 
to produce those commodities. While the decisions to "go open source" were made 
at the tops of those companies, in every case they also involved ratification of deci 
sions already made by the companies own engineers. 

You can still build platforms today, of course, but for practical considerations, it only 
makes sense to build them on top of open source infrastructure. Amazon and Google 
are familiar platform businesses (one for retailing, the other for advertising), built on 
cheap or free open source building materials. 

The portfolio of open source building materials now runs to 100,000 or more prod 
ucts, each the project of a hacker or community of hackers, each producing goods, 
the primary purpose of which is to be useful, not just to sell. Because that software is 
useful, and most of it doesn t have a proprietary agenda, whole market categories can 
be opened where once only proprietary platforms and silos grew. 

Take the private branch exchange (PBX) telephone business. In the old days, which 
are now starting to end, companies had to choose corporate phone systems from 
Toshiba, Panasonic, NEC, Nortel, and other manufacturers of closed proprietary 
platforms and silos. Then a small device maker, Digium, released Asterisk, an open 
source PBX. In addition to a vigorous development community, Asterisk attracted 
countless varieties of businesses made possible by its wide-open use value. In the 
long run, far more money will surely be made because of Asterisk than could ever 
have been made by selling Asterisk or even by selling the proprietary PBXes that 
Asterisk now obsoletes. 

So, thanks to open source, the software market is finally growing up. It is becoming 
mature. Its healthy new ecosystem is made possible by countless commodities, grow 
ing more numerous every day. 

There is an important difference, however, between open source commodities and 
those derived from raw materials (like wood or steel) that is harvested or mined. It s 
a difference that will make the new, mature, software marketplace incalculably large. 

The difference is this: open source commodities are produced by creative and 
resourceful human minds. Not by geology, biology, and botany. This means there is 
neither a limit to the number of open source products, nor a limit to the number of 
improvements. 



Foreword: Source Is Everything 



Yet every one of those open source projects is concerned mostly with the improve 
ment of their own products. While they care about how those products interoperate 
with other products, they can t begin to account for all the combined possibilities 
where interoperation is required. That means there is room for businesses to test, 
certify, and support combinations of open source products. 

That s what attracted me into the vast and growing new marketplace opened by a 
growing abundance of open source building materials. Like many industry veterans, 
I didn t see that opportunity until I moved my point of view from the supply side 
that felt threatened to the demand side that felt empowered. 

And I m hardly alone. 

Some will say we re at the beginning of another boom or worse, another bubble. 
Those views are both limited and misleading. Open source has changed the world 
of software into one in which raw materials are literally limitless. Every mature 
industry such as construction, automobiles, computing hardware has experi 
enced hyper-accelerated growth resulting from commoditization of its core build 
ing blocks. But the impact on the software industry has the potential to be far more 
profound, because software is so malleable, so easily shared now, and so increas 
ingly ubiquitous in everyday life. 

So, while open source software, and the commoditization it brings to the software 
industry, may seem a threat today, it is actually ushering in a wave of enormous 
innovation and productivity the impact of which has already reached far beyond 
this industry. 

It comes down to one simple truth: we humans naturally desire to improve our own 
world through building useful tools. In sharing those tools, we ve learned that the 
world around us gets better too. So the idea of open source is as old as civilization 
itself. And our very modern industry is finally realizing the power contained in this 
simple fact: the first source for everything we make is ourselves. 

KIM POLESE, CEO, SPIKESOURCE 



xii X Foreword: Source Is Everything 



XX 
XX 



Acknowledgments 



Chris DiBona: Like its predecessor, the publication of Open Sources 2.0 represents 
the work of dozens of people both inside and outside of O Reilly. It was my pleasure 
and privilege to work with Mark Stone again, and Danese Cooper was invaluable to 
the creation of the book. The inspiration behind the international section was hers, 
and she should be called out for that. In the first Open Sources I noted that Mark had 
said that "a book could be written about how this book was written." And while this 
one s creation was hardly as dramatic, it was no less challenging. 

Special thanks to Mike Hendrickson, our O Reilly editor, who made it almost too easy. 
Thanks to Tim, Rael, and Nat for the books, conferences and knowledge that your 
company has crammed into my brain. Keep up the great work! Additionally, thanks to 
the folks at Google who allowed me the spare cycles to produce the book so, thanks 
Bill Coughran! I d also like to extend my love to Denise and Neil Kruse, my parents 
Bennie and Cynthia, and especially my sister Trish we miss you. 

Finally, in the last Open Sources, I dedicated it to my patient girlfriend Christine, and 
now I dedicate this book and all my life s works to my wife Christine and our daugh 
ter Frannie. I love you more than I can say. 

Danese Cooper: Thanks to my family Joey, Adi, Zoe & Marie who have put up 
with lots of absences as I ve traveled the world meeting open source people. Thanks 
also to my friends, especially Brian Behlendorf and Tim O Reilly, and to all essay 
writers for agreeing that there should be an update on Open Sources. Thanks to my 
employers at Sun, and now Intel, for giving me space to work on the book, and to 

X nil 



my colleagues at the Open Source Initiative for including me in the work. And 
finally, thanks to my co-editors, Chris and Mark, who had all the experience from 
creating the first book and generously shared it. 

Mark Stone: This book is dedicated to my wife Karen and my son Alex; may your 
future always be open. Looking at the list of contributors here, I realize the three of 
us really do stand on the shoulders of giants and are privileged to facilitate what 
they have created. Several people at O Reilly deserve special praise: Tim O Reilly, 
for having the vision to recognize that the time was right for this book; Mike Hen- 
drickson, who waited so patiently for the final manuscript; and Jamie Peppard, 
Marlowe Shaeffer, Audrey Doyle, and Rob Romano, who had the difficult produc 
tion task of turning a wide range of formats and styles into a unified whole. My co- 
editors, Chris and Danese, have been invaluable and inspiring colleagues through 
out this whole process. Finally, I d like to thank xeno42, elcoronel, and beret for 
the example they set and the education they ve given me; you guys live the ideal 
every day that the rest of us can only talk about. 



xiv X Acknowledgments 







List of Contributors 



Danese Cooper has a 15-year history in the software industry, and has long been an 
advocate for transparent development methodologies. Danese worked for six years at 
Sun Microsystems, Inc. on the inception and growth of the various open source 
projects sponsored by Sun (including OpenOffice.org, java.net, and blogs.sun.com). 
She was Sun s chief open source evangelist and founded Sun s Open Source Pro 
grams Office. She has unique experience implementing open source projects from 
within a large proprietary company. She joined the Open Source Initiative (OSI) 
board in December 2001 and currently serves as secretary and treasurer. As of March 
2005, Danese is with Intel to advise on open source projects, investment, and sup 
port. She speaks internationally on open source and licensing issues. 

Chris DiBona is the open source programs manager for Mountain View, California- 
based Google, Inc. Before joining Google, Chris was an editor/author for the popular 
online web site, Slashdot. He is an internationally known advocate of open source 
software and related methodologies. Along with Mark Stone and Sam Ockman, he 
edited the original Open Sources. He writes for many publications and speaks interna 
tionally on software development and digital rights issues. His home page and blog 
can be found at http://dibona.com. 

Mark Stone has made a career of studying collaborative communities. As a univer 
sity professor with a Ph.D. in philosophy of science, he has studied and published on 
the disruptive community conditions that create scientific revolutions. More recent 



work has involved the open source community, as editor for Morgan Kaufmann Pub 
lishers covering operating systems and web technology, then as executive editor for 
open source topics at O Reilly, and as the editor-in-chief of the Journal of Linux Tech 
nology. While at O Reilly he co-edited, with Chris DiBona and Sam Ockman, the 
seminal Open Sources in 1999. For the last six years he has worked with various dot 
coms on tools and practices for collaboration and online community building, 
including as part of the executive team managing top-tier technology sites such as 
Slashdot (3.5 million page views per day served), and SourceForge.net (1 million 
registered users). As director of product development for ManyOne Networks, he is 
currently working on the next evolution of online community, leveraging 3D envi 
ronments and new tools for knowledge management. Mark holds a Ph.D. in philoso 
phy of science from the University of Rochester, and earned his B.A. in philosophy 
from the University of Maryland. Mark can be reached at mark.stone@gmail.com. 

Robert Adkins is cofounder of Technetra, a Silicon Valley software company which 
implements and deploys large-scale software projects specializing in open source 
solutions. Robert has more than 20 years of experience in the information technol 
ogy industry, having led products and services groups at Apple Computer, IBM, BBN 
Communications, and Litton/PRC. He has an M.S. in computer science from Johns 
Hopkins University. He has published in many technology magazines and journals 
including Linux Journal, LINUX For You, the Journal of the ACM, and Government 
Computer News and speaks frequently at international technology events. Robert can 
be reached at radkins@technetra.com. 

Jeremy Allison is one of the lead developers on the Samba Team, a group of program 
mers developing an open source Windows-compatible file and print server product for 
Unix systems. Developed over the Internet in a distributed manner similar to the Linux 
operating system, Samba is used by multinational corporations and educational estab 
lishments worldwide. Jeremy handles the coordination of Samba development efforts 
worldwide and acts as a corporate liaison to companies using the Samba code commer 
cially. He works for Novell, which funds him to work full time on improving Samba 
and solving the problems of Windows and Linux interoperability. 

Matt Asay has been involved with open source since 1999, and has made a fetish of 
understanding novel ways to monetize open source software. To this end, Matt 
founded the Open Source Business Conference as a place to aggregate and cluster 
people much more intelligent than he to figure out promising open source business 
strategies; cofounded Novell s Linux Business Office and helped to kick-start the 
company s growing Linux business; served as an entrepreneur-in-residence at 
Thomas Weisel Venture Partners, dedicated to finding and developing open source 
investment opportunities; and ran embedded Linux startup Lineo, a network and 
communications business, until its acquisition by Motorola in 2002. Matt speaks and 
publishes frequently on open source business strategy, and consults frequently for 
several open source startups and venture capital firms. 

xvi X List of Contributors 



Matt is currently the general manager at Volantis Systems, where he manages the 
company s growing business with content providers (like eBay, Disney, and Yahoo!). 
He is applying the lessons of open source to the fragmented mobile world, hoping it 
will yield the same standardization and opportunity in mobile/embedded that open 
source did for the server world. 

Matt holds a J.D. from Stanford, where he worked with Professor Larry Lessig on 
analyzing the GPL and other open source licenses. 

Mitchell Baker has been the general manager of the Mozilla project (officially known 
as its Chief Lizard Wrangler) since 1999. The Mozilla project strives to create great 
software and maintain choice and innovation in key Internet client applications, such 
as its flagship Mozilla Firefox and Mozilla Thunderbird products. It is one of the larg 
est open source software development projects in existence. The Mozilla project 
combines dedicated volunteers, a set of paid contributors, and its own flavor of engi 
neering management. 

With the formation of the Mozilla Foundation in 2003, Mitchell also took on the role 
of president of the Mozilla Foundation. Mitchell is also a board member of the Open 
Source Applications Foundation, which is developing a new-style personal informa 
tion manager, known as Chandler. 

Jeff Bates brings many years of strategic management and editorial leadership to the 
Open Source Technology Group (OSTG). As vice president of editorial operations 
and executive editor of Slashdot, Jeff is responsible for setting strategy and integra 
tion for the company s business development partnerships and for driving new site 
and product development, and for fun he helps manage strategic story editing and 
placement for the leading proprietary news site, Slashdot. While at Slashdot, Jeff has 
been responsible for the site winning several industry awards including a Webby 
People s Voice Award for Community, as well as Yahoo ! s "Top 100" Best of the 
Internet Award. Slashdot has also been cited by The Washington Post, Brill s Content, 
TIME, USA Today, Rolling Stone, and other industry-leading publications as one of the 
most innovative and important sites for the technical community. 

Jeff has spoken at numerous academic institutions and industry-leading conferences 
and events, including MIT, LinuxWorld, Worcester Polytechnic Institute, Northern 
Michigan University, Sun Developers Group, the Asian Open Source Symposium, 
Conference of Australian Linux Users, O Reilly s p2p Conference, and the University 
of Michigan. He s also a member of the Open Source Advisory Panel for the U.S. gov 
ernment. Jeff holds a bachelor s degree in history from Hope College. 

Jesus M. Gonzalez-Barahona teaches and conducts research at the Universidad Rey 
Juan Carlos, Mostoles (Spain). He started to work in the promotion of libre software 
in 1991. Since then, he has carried on several activities in this area, including orga 
nizing seminars and courses and participating in working groups on libre software, 



in Spain and throughout the rest of Europe. Currently he collaborates with several 
libre software projects (including Debian) and associations, writes in several media 
about topics related to libre software, and consults for companies and public admin 
istrations on issues related to their strategy on these topics. His research interests 
include libre software engineering, and in particular, quantitative measures of libre 
software development and distributed tools for collaboration in libre software 
projects. In this area, he has published several papers, and is participating in some 
international research projects (visit http://libresoft.urjc.es for more information). He is 
also one of the promoters of the idea of a European masters program on libre soft 
ware, and has specific interest in education in that area. On the personal side, he 
enjoys living, sleeping, and staying with his family (and not in that order). 

Andrew Hessel is a biologist and programmer who has worked at the interface of 
industry and academia to facilitate scientific initiatives, usually in the area of genom- 
ics. He is fascinated by the functional similarities between electronic and biological 
systems, and the lessons that can be learned by comparing them. Andrew lives in 
Toronto, Canada, with his wife Stephanie, and works to advance collaborative breast 
cancer research and therapeutic development. 

Pamela Jones is the founder of Groklaw (http://www.grofelaw.net), an experiment in 
applying open source principles to the field of legal research. Groklaw is also an 
independent journalistic voice, covering legal news stories from the point of view of 
the Free and Open Source (FOSS) community. Groklaw is also an anti-FUD web site. 
It has focused heavily on the SCO litigation, because the community is, after all, 
while not a direct party to any of the lawsuits, directly interested in and affected by 
the outcome, since it is their code and their community that is under attack. For that 
reason, Pamela found it is both natural and appropriate that Groklaw try to contrib 
ute to a positive outcome. 

Eugene Kim is the cofounder and principal of Blue Oxen Associates, a think tank 
and consultancy focused on improving collaboration. He has developed collabora 
tive strategies for a number of organizations, focusing especially on interorganiza- 
tional collaboration and collaborative learning. His research centers around identify 
ing patterns of collaboration across different domains (with a special focus on open 
source communities) and on improving the interoperability of collaborative tools. 
Previously, Eugene worked closely with computer pioneer Doug Engelbart, who cur 
rently serves on the Blue Oxen Associates advisory board. He received his A.B. in his 
tory and science from Harvard University. 

Ben Laurie is a founding director of the Apache Software Foundation, a founder and 
core team member of OpenSSL, the author of Apache-SSL, director of security for 
The Bunker Secure Hosting Ltd.l, coauthor of Apache: The Definitive Guide, and a fre 
quent writer of articles and papers on security, cryptography, and anonymity. You 
can find his web page at http://www.apache-ssl.org/ben.html. 

xvin X List of Contributors 



Louisa Liu is business development manager of the Channel Software Operation 
(CSO) in Intel China Ltd. She is responsible for strategic business development in 
China supporting the CSO. Louisa earned bachelor and master s degrees with hon 
ors in computer science from Fudan University and Tongji University. 

Ian Murdock is cofounder, chairman, and chief strategist for Progeny. He is cen 
trally involved in defining Progeny s technology and business strategies, and in estab 
lishing and maintaining key relationships with customers and partners. Ian has more 
than 10 years of experience in the software industry. He played an instrumental role 
in the transition of Linux from hobby project to mainstream technology by creating 
Debian, one of the first Linux-based operating systems, called distributions. Ian led 
Debian from its inception in 1993 to 1996, building it from an idea to a worldwide 
organization of more than 100 people in less than three years. 

Today Debian is one of the most popular Linux platforms in the world, with mil 
lions of users worldwide. Debian is also widely considered one of the most success 
ful and influential open source projects ever launched: more than 1,000 volunteers 
in all parts of the world are currently involved in Debian development, and the 
founding document of the open source movement itself was originally a Debian posi 
tion statement. 

An Indiana native, Ian holds a B.S. in computer science from Purdue University and 
was a founding director of Linux International and the Open Source Initiative. 

Russ Nelson is a computer programmer and a founding board member of the Open 
Source Initiative. He is best known for his packet driver collection, begun while at 
Clarkson University in 1988. He started Crynwr Software to support his open source 
software, Freemacs (currently used by FreeDOS) and Painter s Apprentice (a Mac 
Paint clone), and went full time with the packet driver collection in 1991. He has 
been making a living from open source support ever since then. His politics are both 
left and right of center, as he is a pacifist Quaker and a member of the Libertarian 
Party of the United States. 

Michael Olson is president and chief executive officer of Sleepycat Software. 
Michael, one of the original authors of Berkeley DB, is a technology industry veteran 
with more than 20 years of experience in engineering, marketing, sales, and business 
management. He was named president and CEO of Sleepycat in 2001 after serving as 
vice president of sales and marketing. Prior to Sleepycat, he served in technical and 
business management positions at database vendors Britton Lee, Illustra, and Infor 
mix. He holds B.A. and M.A. degrees in computer science from the University of Cal 
ifornia at Berkeley. 

Tim O Reilly is founder and CEO of O Reilly Media, thought by many to be the best 
computer-book publisher in the world. In addition to publishing pioneering books 
such as Ed Krol s The Whole Internet User s Guide &> Catalog (selected by the New York 



List of Contributors X xix 



Public Library as one of the most significant books of the 20th century), O Reilly Media 
has also been a pioneer in the popularization of the Internet. O Reilly s Global Network 
Navigator site (GNN, which was sold to America Online in September 1995) was the 
first web portal and the first true commercial site on the World Wide Web. 

O Reilly Media continues to pioneer new content developments on the Web via its 
O Reilly Network affiliate, which also manages sites such as Perl.com and XML.com. 
O Reilly s conference arm hosts the popular Perl Conference, the Open Source Soft 
ware Convention, and the O Reilly Emerging Technology Conference. 

Tim has been an activist for Internet standards and for open source software. He has 
led successful public relations campaigns on behalf of key Internet technologies, 
helping to block Microsoft s 1996 limits on TCP/IP in NT Workstation, organizing 
the "summit" of key free software leaders where the term open source was first widely 
agreed upon, and, more recently, organizing a series of protests against frivolous soft 
ware patents. Tim received Infoworld s Industry Achievement Award in 1998 for his 
advocacy on behalf of the open source community. 

Tim graduated from Harvard College in 1975 with a B.A. cum laude in classics. His 
honors thesis explored the tension between mysticism and logic in Plato s dialogs. 

Gregorio Robles is a teaching assistant and a Ph.D. candidate at the Universidad Rey 
Juan Carlos in Madrid, Spain. His research work is centered in the empirical study of 
libre software development from a software engineering point of view. He has 
authored or coauthored many papers that were presented at both academic and com 
munity conferences, and has developed or collaborated in the design of programs to 
automate the analysis of libre software. He has also been involved in the seminal 
European Union FLOSS study and survey on libre software developers, the CALIBRE 
coordinated action to foster libre software development in Europe, and the FLOSS- 
World study which looks at libre software development worldwide, all of them 
financed by the European Commissions 1ST program. 

Larry Sanger was the chief organizer/architect of the Wikipedia encyclopedia project 
in its first year, as well as of the now-moribund Nupedia encyclopedia project. Since 
2000 he has thought and written about the best ways to develop a collaboratively 
built online encyclopedia. He is now working on that problem, among others, for the 
ambitious Digital Universe project as its director of distributed content programs. His 
Ph.D. (2000) from Ohio State University is in philosophy, with concentrations in 
epistemology and early modern philosophy, and his B.A. in philosophy is from Reed 
College in Portland, Oregon. He taught a wide range of philosophy courses off and 
on between 1992 and 2005 for Ohio State University and nearby institutions. He 
also plays Irish traditional music on the fiddle and has taught that too, off and on 
since 1997. 



xx * * List of Contributors 



Sunil Saxena is senior principal architect in the Software and Solutions Group (SSG) 
at Intel Corporation. SSG is responsible for operating system enabling on Intel archi 
tecture products. Sunil received his Ph.D. in computer science from the University of 
Waterloo and received his B. Tech. in electrical engineering from Indian Institute of 
Technology, Delhi, in 1975. 

Doc Searls is a writer and speaker on topics that arise where technology and busi 
ness meet. He is the senior editor of Linux Journal, the premier Linux monthly and 
one of the world s leading technology magazines. He also runs the new Doc Searls IT 
Garage, an online journal published by Linux Journal s parent company, SSC. He is 
coauthor of The Cluetrain Manifesto: The End of Business as Usual, a New York Times, 
Wall Street Journal, Business Week, Borders Books, and Amazon.com bestseller. (It was 
Amazon s #1 sales and marketing bestseller for 13 months and sells around the world 
in nine languages.) He also writes the Doc Searls weblog. J.D. Lasica of Annenberg s 
Online Journalism Review calls Doc "one of the deep thinkers in the blog movement." 
Doc s blog is consistently listed among the top few blogs, out of millions, by Techno- 
rati, Blogstreet, and others. 

Wendy Seltzer is an attorney and special projects coordinator with the Electronic 
Frontier Foundation, where she specializes in intellectual property and free speech 
issues. In the fall of 2005, she will be at Brooklyn Law School as a visiting professor 
of law. As a fellow with Harvard s Berkman Center for Internet <Sr Society, Wendy 
founded and leads the Chilling Effects Clearinghouse, helping Internet users to 
understand their rights in response to cease-and-desist threats. Prior to joining EFF, 
Wendy taught Internet law as an adjunct professor at St. John s University School of 
Law and practiced intellectual property and technology litigation with Kramer Levin 
in New York. Wendy speaks frequently on copyright, trademark, open source, and 
the public interest online. She has an A.B. from Harvard College and a J.D. from Har 
vard Law School, and occasionally takes a break from legal code to program (Perl). 

Sonali K. Shah is an assistant professor at the University of Illinois at Urbana-Cham- 
paign. Her research focuses on the creation and maintenance of novel organizing 
innovation communities that support innovation development and diffusion. She 
studies innovation communities in fields as diverse as open source software, sports 
equipment, and medical products. A second stream of work examines the processes 
underlying the formation of new industries and product markets. Previously she 
worked at Morgan Stanley & Co. and McKinsey & Co. She holds degrees in biomed- 
ical engineering, finance, and management. She is a graduate of the University of 
Pennsylvania and the Massachusetts Institute of Technology. 

Alolita Sharma is cofounder and CEO of Technetra, a Silicon Valley software com 
pany which implements and deploys large-scale software projects specializing in 
open source solutions. Alolita has more than 14 years of experience in the informa 
tion technology industry, having engineered and led services groups at IBM, MCI 



List of Contributors X xxi 



Worldcom, Intelsat, and SWIFT. While pursuing the Ph.D. program in computer sci 
ence at George Washington University (GWU), she concentrated on networking, 
security, and parallel computing. She received an M.S. in computer science also from 
GWU. She speaks at technology forums and has published in many technology mag 
azines and journals including Linux Journal, Linux Gazette, and is a monthly colum 
nist for India s only open source magazine, LINUX For You. She is a proponent of 
Linux and open source software in India. Alolita can be reached at as@technetra.com. 

Bruno Souza is a senior consultant at Summa Technologies. He helps large compa 
nies to successfully use and develop open source products and projects. Bruno is 
president of Soujava, Brazil s largest Java User Group, where he has led the group s 
Javali Project, an ambitious umbrella project that hosts 10 large open source projects. 
Javali, which includes a project to create an open source Java runtime, is targeted to 
bring software development into Brazil s open source discussions. Bruno also co- 
authored Soujava s Open Source Manifest, which discusses open source and open 
standards as the way to correctly apply and succeed with open source in Brazil. The 
document positively influenced the adoption of open source in Brazil. Bruno is a 
member of the Management Board ofjava.net, one of the largest open source hosting 
sites for Java developers, where he leads the World Wide Java User Group Commu 
nity, is an activist for the creation of open source-compatible implementation of Java 
standards, and is an active participant in several Java open source projects. 

Stephen R. Walli has worked in the IT industry since 1980 as both customer and 
vendor. He is presently the vice president of Open Source Development Strategy for 
Optaros. Stephen is responsible for architecting and managing Optaros s relation 
ships with the open source community. Most recently, Stephen was a business devel 
opment manager at Microsoft on the Windows Platform team, where he operated in 
the space between community development, standards, and intellectual property 
concerns. While at Microsoft, he also worked on the Rotor project (Shared Source 
CLR), and started as the product unit manager for Interix in Services for Unix. 

Prior to Microsoft, Stephen was the vice president of R&D and a founder at Softway 
Systems, Inc., a venture-backed startup that developed the Interix environment to re- 
host Unix applications on Windows NT. Stephen has also worked as an indepen 
dent consultant for X/Open, SunSoft, UNISYS, and the Canadian government. He 
was once a development manager at Mortice Kern Systems, and a systems analyst at 
Electronic Data Systems. 

Stephen was a longtime participant and officer at the IEEE and ISO POSIX standards 
groups, representing both USENIX and EurOpen (E.U.U.G.), and has been a regular 
speaker and writer on open systems standards since 1991. 

He blogs at http://stephesblog.blogs.com, and occasionally podcasts from http.7/ 
stephenrwalli.users.blogmatrtK.com/poacasts. 



xxii * C List of Contributors 



Steven Weber, a specialist in international relations, is an associate with the Berke 
ley Roundtable on the International Economy (BRIE) and the International Com 
puter Science Institute, and affiliated faculty of the Energy and Resources Group. His 
areas of special interest include international politics, and the political economy of 
knowledge-intensive industries. 

Steven went to medical school at Stanford and then earned his Ph.D. in the political 
science department at Stanford. In 1992, he served as special consultant to the presi 
dent of the European Bank for Reconstruction and Development in London. He has 
held academic fellowships with the Council on Foreign Relations and the Center for 
Advanced Study in the Behavioral Sciences. He is a member of the Global Business 
Network in Emeryville, California, and actively consults with government agencies 
on foreign policy issues, risk analysis, strategy, and forecasting. 

Boon-Lock Yeo is currently director of the ICSC (Intel China Software Center) of the 
SSG (Software and Solutions Group) in Intel China Ltd. He received his Ph.D. in 
electrical engineering from Princeton University and a BSEE from Purdue University. 
He received an IEEE Transactions Best Paper Award in 1996, has published more 
than 40 technical papers, and holds 25 U.S. patents. 



List of Contributors * * XXHI 



XX 
XX 



Chris DiBona, Danese Cooper, 
and Mark Stone 



Introduction 



Midnight comes to the Nevada desert, and nothing is visible but a line of taillights 
ahead and a line of headlights behind. 

"You ll see the lights from Gerlach first, and then Black Rock City," the driver says to 
the passenger. They drive another 20 minutes in silence before the highway crests a 
ridge line. On the horizon, the blackness is broken by a band of multicolored light. 

"Is that Gerlach?" the passenger asks. 

"No." The driver points to a small, dim cluster of yellow lights in the middle dis 
tance. "That s Gerlach. Way out there, those lights are Black Rock City. We re still 
about an hour away." 

The passenger ponders this for a moment, then asks, "An hour? How big is it?" 

"For one week each year, Black Rock City is the fourth largest city in Nevada. Popu 
lation 30,000, give or take." 

Gerlach rolls by, with its one bar, one gas station, and one motel. The caravan of cars 
bunches up after Gerlach. Then they turn off the highway, rumbling over the packed 
mud playa of the Black Rock Desert. Lights, neon, and thousands of RVs spread out 
before them. Music and drums especially drums can be heard in the distance. At 
the gate they re approached by someone who looks like a transplant from Mardi 
Gras: face paint, bright-colored suit, and a carnival hat. He checks their tickets and 
flashes them a big grin. 



XXV 



"Welcome to Burning Man! 



When the original Open Sources was published in 1999, it served mainly as an affir 
mation that open source existed. The book brought together the leading voices in 
open source, demonstrating that we were a community, that we were indeed a move 
ment to be taken seriously. To put that time in context: 

Microsoft had, only a year earlier, leaked the "Halloween Memo," its first semi- 
public acknowledgment that open source was a competitive threat. 

IBM had provided some initial backing for Apache, but had yet to announce its 
$1 billion Linux initiative. 

Linux was in only the 2.2 stage of kernel development. 

SourceForge.net was a relatively new site with only a few hundred projects 
hosted. 

The mainstream press could not separate rising interest in open source from dot-corn 
bubble hype. The media took the surface ideas of Eric Raymond s The Cathedral & the 
Bazaar and created a caricature of legions of hobbyist programmers distributed 
across the globe, competing against the technology Goliaths of the day. That picture 
bore no more resemblance to reality than the tale of King Arthur does to the histori 
cal Middle Ages. Yet like any good mythology, it serves as a useful point of depar 
ture for understanding the real history from which it arose. 

Today open source is an accepted fact of business life, with many companies engaged 
in core open source business models (Sleepycat, MySQL) or significant hybrid models 
blending open source and proprietary software (IBM, Novell, Red Hat). Many compa 
nies have striven to incorporate open source development into their range of software 
development practices even Microsoft has projects hosted on SourceForge.net. How 
much do we really understand about the dynamics of open source software develop 
ment or the communities that stand behind those projects? 

The essays presented in this volume take a major step forward in our understanding 
since 1999, when the original Open Sources was published. 



Burning Man is approaching its 20th anniversary. Conceived and inspired by Larry 
Harvey, it began in 1986 as a gathering of dozens of participants at Baker Beach in 
San Francisco. The centerpiece of the event was then, as it is now, the construction 
of a wooden effigy of a man, which is burned in celebration. 

Celebration of what? The answer to that question may differ for every participant. 



xxvi ** Introduction 



The event was originally timed with the summer solstice it s now held the week 
leading up to Labor Day and has always had a pagan, tribal feel to it. Participants 
began bringing their own art projects, many of which were also burned in celebra 
tion at the climax of the event. By 1990 the gathering numbered in the hundreds, 
and even the unusually tolerant San Francisco police made it clear that the event 
needed to find another venue. 

Burning Man then moved to its current location the Black Rock Desert an empty 
stretch of Nevada desert on federal Bureau of Land Management land, roughly two 
hours north of Reno. The extreme remoteness and the harsh environment have become 
an indelible part of the event. To be there, you have to really want to be there. 



Mitchell Baker makes clear in her essay that part of the strength of the Firefox commu 
nity is its size. Thousands of people have contributed to Firefox, a community of con 
tributors larger than the core project leaders can really envision. Firefox seems very 
much like one of those mythical "legion of programmers" projects that comes to mind 
when people think of the metaphor suggested by Eric Raymond s The Cathedral & the 
Bazaar. 

Yet the open source development model remains the most enigmatic aspect of the 
open source community. One striking and unexpected outcome of the years since 
the original Open Sources is how little technology companies have been able to lever 
age the open source development process. Projects such as Linux and Apache have 
had world-changing success, yet no commercial software company has been able to 
replicate this development process for its own products or its own success. AOL, for 
example, has never figured out how to integrate the Mozilla/Firefox developer com 
munity into its product development process. Sun has struggled to open up both 
Java and Solaris, and the jury is very much out on the success of those projects. 

Read the essays here by Chris DiBona and Jeremy Allison and you will see how little 
proprietary software development differs from open source software development. 
The differences their essays suggest are subtle: an emphasis on knowledge reuse, not 
just code reuse; a recognition that open standards matter; and that architecture needs 
to be created with openness in mind. Why, though, are these and other open source 
lessons so hard for commercial companies to use? 

The real paradox is as old as Fred Brooks classic, The Mythical Man-Month. In this 
work, Brooks formulates what has become known as Brooks Law: that while the 
amount of programming work completed increases linearly as the number of pro 
grammers increases, the complexity of a project increases as the square of the num 
ber of programmers. The result is that large programming teams fail to reduce the 
time to project completion. The rationale is that the number of communication inter 
faces, which is roughly equivalent to the amount of coordination effort the project 
requires, increases geometrically as more people are added to the project. 



Introduction X xxvn 



Brooks Law appears to set a fundamental limit on the optimal size of programming 
teams and a rather small limit at that. Empirical evidence supports Brooks Law. 
For example, since its inception SourceForge.net has maintained very close to a 10:1 
ratio of registered users to registered projects, suggesting that open source develop 
ment projects seldom have more than 10 active developers. 

What are we to make, then, of the thousands of Firefox contributors? The key is to 
recognize that they are not a homogeneous mass of contributors, and Firefox is not a 
monolithic piece of software. In fact, the design is highly modular, enabling small 
teams to work on separate components of the code without interfering with each 
other. By far the most common characteristic open source projects have is a highly 
modular design. That design architecture is more a choice of social engineering than 
technical engineering, however. In the original Open Sources, Linus Torvalds com 
mented that he started with a monolithic kernel design for Linux because he knew it 
would provide higher performance. Only later was Linux s distinctive system of load 
able kernel modules developed, and it was developed as much out of a project man 
agement need as anything else. Could Apache provide higher performance as a 
purely monolithic piece of code? Probably. But the development process would 
become unmanageable. The original release of Mozilla suffered terribly because it 
was a monolithic piece of code. Firefox is the result of years of rearchitecting to 
achieve a modular, and thus manageable, design architecture. 

Once it s clear that programming teams must necessarily be small, and that modular 
ity is driven by communication and management needs more than by engineering 
needs, the structure of the open source community makes a lot more sense. The 
"bazaar" looks less like a bustling, homogenous mass and more like a structured 
community. More than anything, it resembles a tribe. 

In fact, there is very little of Brooks Law that is unique to software development. Any 
creative, collaborative knowledge enterprise faces the same constraints, and as a 
result, many collaborative communities adopt the same tribal structure. 



Traditionally Black Rock City is laid out in a half-circle, with a grid of "streets," some 
of which form the spokes of the wheel, and others of which form concentric rings 
spreading from the center. At the very center is the Man, 40 feet of elegantly assem 
bled wood, awaiting his night of conflagration at the end of the weeklong event. 

A "burner" approaches a small tent encampment at the corner of 7 O clock and Justice. 
Out of the back of a pickup truck, two men from the encampment haul bags of sand 
and large rocks painted in black-light colors. They place the rocks inside a wooden 
sandbox and pour sand into it. Above them, a hand-painted sign reads "Reflections in 
Sand." 

"So, what s your project?" asks the passerby. 



XXVIII 



One of the men looks up, squinting under the relentless desert sun. "We re making a 
Zen garden," he replies. He gestures toward a black light and generator lying nearby 
next to a couple of small, handmade wooden rakes. "Only this will be one you can 
enjoy at night." 

"Where are you guys from?" 

"San Francisco." 

"You drove 350 miles to bring sand to the desert? Cool." 

Burning Man is a participatory event. According to its mission statement (http:// 
\vww.bumingmanxom/whatisburningman/about_burningman/mission.html): 

Our intention is to generate society that connects each individual to his or her 
creative powers, to participation in community, to the larger realm of civic life, 
and to the even greater world of nature that exists beyond society. 

While not everyone who comes brings an art project, the expectation is that projects 
will be interactive in nature and that no one is there simply to observe. All are there 
to participate. Burning Man has no "audience." While the event is not a political 
gathering, many projects have a definite message connecting to that "larger realm of 
civic life," and to encourage this, each year Burning Man has an overall theme. 
Recent themes have included "The Vault of Heaven," "Beyond Belief," and "The 
Floating World." 

Several other ideas underpin the spirit of Burning Man. Black Rock City LLC has had 
a delicate relationship with the Bureau of Land Management over the years, striving 
to show that the event is a positive part of the area. A key part of this is Burning 
Man s "leave no trace" philosophy. Art projects are not permanent, but are designed 
to be temporary, something can be removed or destroyed at the end of the event 
without leaving trash behind. Volunteers spend weeks after each event restoring the 
Black Rock Desert to its pre-event conditions. The "leave no trace" policy is not just a 
practical matter, though: it is also a good philosophical fit with the civic aspect of the 
event and with the fragile sense of the temporary that has been at the heart of the 
event since the first Burn. 

Burning Man is also basically a nocturnal event. At roughly 6,000 feet elevation, the 
Black Rock Desert air is thin and dry. Daytime temperatures can hover around 100 
degrees, so dehydration is the most common condition treated at the infirmary. As 
the sun goes down each day, people pause at whatever they are doing, look toward 
the craggy western hills, and cheer. Drums begin to beat. Music begins to play. Black 
Rock City comes alive. 

Temporary, participatory, and nocturnal: those are the key elements of a Burning 
Man art project. 



XXIX 



As puzzling as how open source projects organize themselves is why. To the casual 
outside observer, it appears that open source developers spend enormous amounts of 
time developing software that, in the end, they are simply going to give away with 
out the prospect of compensation in return. While open source certainly has an 
altruistic side, altruism is neither the only nor the most important motivation. 

First, one must realize that many open source projects started out of a developer s 
desire to solve an immediate problem. Linux was born of Linus Torvalds desire to 
have a development platform on his PC at home. Apache came together from a group 
of people who had been relying on the NCSC web server and wanted to continue its 
development after NCSC stopped maintaining it. 

Read Sonali Shah s essay, and you ll understand that the pattern here extends far 
beyond software development. Many consumer communities have come together 
collaboratively to innovate the products they consume, often when producers fail to 
produce innovation on their own. Software companies created just such a stagnant 
environment, out of which Linux and the rest of open source software was born. In 
this sense, the initial catalyst was quite selfish: developers wanted the software that 
companies were unwilling or unable to produce, so open source developers created it 
to "scratch their own itches." 

What started from largely selfish motivations has evolved into something quite com 
plex. In Steve Weber s essay, we get a clear analysis of just how complex the gover 
nance structures and processes of this community have become, as well as an intrigu 
ing view of where these enabling governance structures might foster collaborative 
communities in other endeavors. Andrew Hessel provides in his essay a very tangi 
ble example of where open source ideas are taking hold in an entirely new realm. 

Lurking in the background, though, is still a question of motivation. If the inspiration 
for Linux was a selfish one, why make the choice to give the result away, and further 
more to do so under open source terms? How does selfishness become altruism? 

The apparent paradox rests on the assumption that acts of charity necessarily con 
flict with acts of self-interest. From the point of view of a modern market economy, it 
often appears that charity and self-interest do conflict. What drives the open source 
developer, however, is clearly self-interest even if it is based on an older notion of 
self-interest not easily captured by modern market economics. 

The answer to the paradox lies in the reputation game played within the open source 
community. After all, monetary compensation is only a means to an end; it is a 
means of providing survival resources. Yet monetary compensation is not the only 
means to that end. While open source luminaries such as Linus Torvalds and Brian 
Behlendorf may not have the personal wealth of fortunate dot-commers, neither will 
ever lack for gainful employment. They have sufficient reputations based on their 
open source accomplishments to always be able to earn a living from their expertise. 



XXX 



Consider another fact: the largest age group among open source hackers is college 
students and graduate students: those under 25 for whom gainful employment is not 
an immediate issue, but one that certainly looms in their thoughts and plans. Because 
they are students, we can assume that they have the immediate survival resources 
needed for one to become a student in the first place. However, we can also assume 
that those survival resources are finite. Securing future survival resources is very 
much a part of the agenda of a student, indeed one of the main reasons for becom 
ing a student in the first place. While a degree may provide a measure of that future 
security, a degree is not the exclusive means to that end: reputation as a creator of 
good code may provide that future security as well as or better than a degree. 

Once this separation is made between monetary compensation and survival 
resources, we can see that there are historical precedents for this kind of behavior, 
and that there is a social model that loosely fits that of open source hacker culture. In 
Western civilization, we can look to medieval Europe, when nomadic groups like the 
Franks and the Vikings had settled into the agrarian-based feudal system. The mature 
feudal system of the 13th and 14th centuries has some interesting and instructive 
social structures; we ll focus on the concept of chivalry. 

The good knight adhered to a code of behavior that transcended the laws of any par 
ticular kingdom and encouraged an attitude with some similarities to the attitude of 
today s hacker: a knight should be humble and should regard himself at the service 
of others, yet he would be judged by his prowess at his trade and would succeed to 
the extent that he could spread the reputation of his prowess. 

Behind shield and visor, and upon adopting a particular set of heraldic emblems, a 
knight took on a kind of persona, creating a public identity that might be quite dif 
ferent from his private identity. There were regular events for testing and publicizing 
one s prowess at arms, such as the tournament or the hunt. There were orders of 
knighthood again, often transcending the boundaries of kingdoms that would 
certify one s prowess. To belong to such an order was an honor; to be of sufficient 
repute to be able to found an order and have other knights flock to it, was perhaps 
the greatest measure of success in the chivalric reputation game. 

While a knight was pan of the nobility, knighthood was a terrible burden financially. A 
suit of armor cost, relative to the medieval standard of living, the equivalent of a brand- 
new Mercedes today. Horses were expensive, and a knight was expected to have sev 
eral. In addition, a knight had to maintain an entourage of squires, pages, and men-at- 
arms. He also officially owed 40 days of service to his feudal lord each campaign sea 
son, 40 days that in reality could often drag into several months. A landed knight with 
a small manor could easily spend any excess capital just to maintain his position; a 
landless knight, or knight errant, would likely live in perpetual debt. 



Introduction * * xxxi 



What motivation, then, would a young man in the Middle Ages have to aspire to 
knighthood? Did one live according to the code of chivalry out of pure selflessness, 
and a desire to serve others? Or was there some more pragmatic force at work? To a 
knight of repute, money was not really important. His lord, or anyone else interested 
in retaining his services, would see that his needs were met, that he had survival 
resources. To flourish, a skilled squire aspiring to knighthood need only hone his 
skills and act to establish and further his reputation. 

This attitude and this kind of behavior seem quite similar to that of the young stu 
dent who is an aspiring hacker. While academia, and academic computer science, is 
a reputation game of its own, what is fascinating about computer science is that there 
is a large body of practitioners that refuse to play this particular reputation game. To 
this latter group, an education really is just a means to an end, and the end is to 
develop the skills necessary to create good code. Some may go far enough to pick up 
a university degree, but many do not, and the degree is clearly secondary to the abil 
ity to code. In other words, one can build a reputation without having to acquire the 
academic pedigree. It is not simply a distaste for academia that fosters this kind of 
behavior. The hacker who is more interested in picking up skills than in picking up a 
degree is often the same hacker who is unwilling to be tied down to a steady job, 
preferring to move from assignment to assignment as a freelancer and consultant. 
These are the knights errant of the open source movement. 

In reality, the medieval knight errant was essentially a mercenary, hardly the noble 
figure portrayed by Malory or Chaucer. The fact that these soldiers were mercenaries 
made chivalry no less important. A mercenary captain had to be trustworthy; his 
word of honor alone was a binding contract. Otherwise, he simply was not employ 
able. These men lived by the chivalric code. What they found was that that code 
alone assured them survival resources. A skilled and honorable mercenary captain 
was never without employment and never lacked for resources. 

Chivalry and pragmatism are not conflicting goals, but that pragmatism can indeed 
be served by chivalry. The mercenary captain lacked money, land, and all other tan 
gible resources. He had only one form of collateral: his reputation. That reputation 
could be maintained only if his behavior was seen to be genuinely honorable. Chiv 
alry, then, was a necessity: it was an essential ingredient in building the only avail 
able collateral that could be parlayed into survival resources. 

From this analogy, we can learn several lessons about today s hackers. First, the open 
source gift culture need not be seen as strictly, or even predominantly, altruistic. 
Pragmatism and altruism are not mutually exclusive. Today s hackers, like the 
knights errant and mercenaries of old, can and do trade in the coin of reputation as a 
means of achieving survival resources. 



xxxn X Introduction 



Second, as a culture matures, the pragmatism becomes more apparent. This was true 
in the Middle Ages. In the high Middle Ages, the era of the crusades, the knight 
errant made a flamboyant pretext of making a gift of his skills and services; one has 
to look below the surface to see a rational exchange of skills for resources in such gift 
acts. By the late Middle Ages, though honor and reputation were still essential, when 
a nobleman retained the services of a mercenary captain, the transaction was explic 
itly and without apology for the mutual benefit of nobleman and captain. 

We see this same trend among today s hackers. While reputation alone has always 
provided survival resources for some, the trend to switch the meme about their activ 
ity from "Free Software" to "Open Source" reflected a maturing shift from altruistic 
pretext to honest pragmatism. 

Today we see a symbiotic balance between the chivalric open source hackers and the 
companies that employ them. In fact, this development is foreshadowed in the events 
Eugene Kim describes in his essay, which concerns the development of the first com 
mercial compilers. Even in the 1950s it was possible for a company such as IBM to see 
the advantages of transforming a competitive relationship into a collaborative one. 

Today a number of prominent open source developers are employed at major tech 
nology teams. Novell transformed itself in a matter of months into a major player in 
the enterprise open source space through the acquisition of Ximian and SuSE. Of 
contributors to this volume, Jeremy Allison works for HP, and Chris DiBona works 
for Google. Neither works at an open source company per se, but both have an 
understanding that it is in their employer s interest that they be allowed time and 
resources to continue working on open source projects. Other contributors here, 
such as Ian Murdock (the "ian" of "debian"), Michael Olson, and Stephen Walli, are 
involved in more purely open source business models. 



Burning Man is, in some sense, a commercial operation. There is a significant admis 
sion charge for the event (more than $200), and the event is run by a limited liabil 
ity corporation. The corporation s main purpose, however, is to sustainably manage 
the event. There is a permit to obtain from the Bureau of Land Management every 
year. There is insurance for the event. There is preparation before and cleanup after 
the event, as well as basic infrastructure, such as sanitation services, that must be 
provided every year. Finally, there is a small paid staff responsible for everything 
from event promotion and organization to informal lobbying efforts with the Depart 
ment of the Interior, Washoe County, and the state of Nevada. 

Once inside the gates, however, participants are forbidden from engaging in mone 
tary commerce. The primary form of commerce is barter. In the spirit of the event, 
barter is as much a pretext for participation as an exchange of goods. It may take the 
form of a scavenger hunt, where admission to an art project requires a ticket stamped 



Introduction *C xxxm 



by several other art projects. It may take the form of a raid by the "Viking Longship" 
art car, which "pillages" camps but always leaves some small gift behind. Or it may 
be in the form of a quiet bar on a back street of Black Rock City that asks only some 
small trinket from the day s events as the price of a drink. 

Unfettered from monetary exchange, however, most denizens of Burning Man gravi 
tate toward a gift economy. Acts of giving range from the mundane to the extravagant: 
the accordion player who serenades those in the porta-potty line with his renditions of 
AC/DC; the massage therapist volunteering her services; the water-gun brigade, spray 
ing people down for a moment of cool relief from the midday sun; or the man who 
brings along a week s supply of dry ice so he can serve cold ice cream every day. 



One of the most ironic developments since the publication of the original Open 
Sources has been the rapid adoption of open source business models by technology 
companies. 

In 1999, the consensus view of the business community was that giving away intel 
lectual property for free was a poor basis for doing business. At that point, Michael 
Olson and Sleepycat Software had been quietly pursuing their dual licensing model 
for Berkeley DB for three years. Now several open source database companies are 
pursuing similar models. 

Sleepycat s approach is an example of the more general business dynamics at work 
behind open source. One of the key effects is commoditization, discussed in different 
aspects in essays by Matt Asay, Ian Murdock, and Stephen Walli. Commoditizing a 
complement to one s core business serves to enhance that business. It brings down the 
cost of entry for customers, thus expanding the potential market size for the core busi 
ness. The key is to have a complementary core that can be monetized. Sleepycat 
achieves this through dual licensing, charging for a proprietary license for customers 
who are unwilling or unable to open their own source. Novell achieves this through a 
hybrid business model, with a service business and a proprietary software business fur 
ther up the "application stack" from its commoditized Linux business. 

Commoditization is not the only benefit. Open source business models lower the 
cost of both sales and marketing. The common fear with any free product is that 
"you get what you pay for." With open source, however, the source code is entirely 
open to inspection so that there are no hidden surprises. Further, the source code 
can be freely redistributed. Several market effects result from this. First, those most 
likely to avail themselves of open source are those with the greatest understanding of 
its benefits namely other software developers who actually have the skill and desire 
to examine source code. Second, the distribution model creates a user community of 
like-minded enthusiasts without intervention or incurred marketing costs by the 



XXXIV 



originating company. Finally, that user base will, at some point, approach the origi 
nating company with a request for additional features, services, or complementary 
software. In other words, by its very nature, open source has very low marketing 
costs that create an inbound sales channel of prequalified leads. 

Open source software companies that exploit this dynamic can thus maintain lower 
overall operating costs, consequently passing on lower prices while still maintaining 
healthy profit margins. All of this accelerates the commoditization process, making a 
well-established open source software product quite difficult to compete against. 

Yet these very business advantages inherent in open source bring us to another 
aspect of the same paradox: why is it so difficult for companies to leverage open 
source as a development model, rather than as a business or marketing model? Con 
sider Sleepycat s dual licensing scheme. The model works only if Sleepycat holds full 
copyright to all of the software in Berkeley DB. Otherwise, it is not permitted to offer 
the second, proprietary license in addition to the open source license. If it must have 
all rights to the software, though, that means that the software must essentially be 
developed in-house. Sleepycat does its own development instead of leveraging out 
side, open source development. 

Perhaps Russ Nelson offers a purer example of an open source business model, one 
where he both develops open source software and leverages the open source devel 
opments of others. The complimentary values Russ Nelson offers are his reputation 
and his expertise, both carefully maintained over the years. The resulting business 
may not be a large one, but it is one where he alone is the master of his own destiny. 
* * * 

Burning Man certainly has the feel of an organic, grass-roots movement. Certainly 
that grass-roots element is part of the dynamic that makes the Burning Man commu 
nity what it is. But simply thinking of "grass roots" makes it too easy to overlook 
what a complex community structure Burning Man has and requires. 

First, there is the structure of Black Rock City itself. Maintaining order in Black Rock 
City is primarily the responsibility of the Black Rock Rangers. They describe themselves 
as a "non-confrontational mediating agency" (see http://www.rangers.org). They are all vol 
unteers, and they have no official authority. They pay admission like everyone else; 
rangering is not a way around the admission price. Further, each ranger is required to 
attend training prior to the event, and each ranger must enter the ranger program with 
the sponsorship of another ranger as mentor. While the rangers occasionally call in 
actual law enforcement, for the most part the rangers are accorded tremendous respect. 

Black Rock City has many of the elements of any other city of 30,000. There is a 
radio station, some years several. There is a DMV (Department of Mutant Vehicles); 
you cannot drive around within Black Rock City or the playa beyond without regis 
tering with the DMV. There is an airport; dozens of attendees routinely fly in for 



xxxv 



Burning Man. And of course, there is a newspaper, the Black Rock Gazette, published 
six times during each Burning Man event. 

Residential areas of Black Rock City have structure as well. Look at a map of Black 
Rock City when you arrive at Burning Man, and you ll see a number of areas along 
the inner circle marked as Theme Camps or Villages. These are areas that are both 
residential and interactive, involving a large number of people working on a com 
mon art project where the residential area itself is the art project. 

The inner circle faces toward the Man, and beyond on the playa is the Burning Man 
"gallery" of art installations. Here is where the larger projects are constructed and the 
larger group events are played out. Typically, there will be an opera and several other 
stage performances. Weddings are common. 

For several years, there was a project called Solaria. It was a scale model of the solar 
system, where not only the distances between objects were proportional, but also the 
size of those objects relative to distance was proportional. Each object was a light 
source, with the sun represented by a small lamp about the size of a bowling ball. On 
that scale, Pluto could be reached only by a three-mile bike ride across the playa. Not 
even the Smithsonian can put on an exhibit of that scale. 



No one has grasped the power of commoditization as quickly as developing nations. 
This is an international arena with general concern over globalization, anxiety about 
domination by American corporations, and fear of Microsoft s monopoly in particu 
lar. Open source has given developing nations a bargaining chip to pressure technol 
ogy companies, especially Microsoft, on price. The natural response has indeed been 
a lowering of prices in countries ranging from Brazil to India. 

Yet the significance of open source goes far beyond commoditization and price pres 
sure. Read the essays here from Jesus M. Gonzalez-Barahona and Gregorio Robles, Alol- 
ita Sharma and Robert Adkins, and Boon-Lock Yeo, Louisa Liu, and Sunil Saxena, and 
you see that open source is really about controlling one s technology destiny. Outside 
the United States, people find it odd that we use the same word,/ree, to mean two very 
different things: with no cost or liberated. The open source community has adopted the 
slogans "free as in beer" versus "free as in speech" to draw attention to the difference. 
Commoditization is all about "free as in beer;" what developing nations care about the 
most is "free as in speech." 

Open source provides greater intellectual property control than proprietary software that 
one does not own. Developing nations want control over the intellectual property on 
which their technology infrastructure depends. What emerges is a different sense of 
ownership from the traditional market economy sense of ownership, one that speaks not 
just to the motivations of policymakers in developing nations, but to the motivations of 
open source developers as well. Think again of chivalry and of our feudal heritage. 



xxxvi 



In a feudal system, a farmer could not own land, nor the harvest from that land. Serfs 
were indentured to the land and were entitled to only a portion of their harvest after 
paying their taxes to the feudal overlord and landowner. 

Technology workers today face an analogous form of servitude. It is almost universal 
practice at technology companies to confront new employees with a hiring agree 
ment that says, among other things, that any and all code and inventions created by 
the employee while in the employ of the company belong to the company; all copy 
rights and patents resulting from these creations must be transferred to the com 
pany. Technology workers may reap the fruits of their creative labor only under 
terms dictated by the company. Our modern notions of intellectual property and 
ownership of it are based on this relationship: that it is fundamentally companies, 
not individuals, that own intellectual property, and that individuals create new intel 
lectual property primarily in the service of companies. 

If open source hackers have one common attitude that ties them together into a com 
munity, it is the rejection of this notion of intellectual property. The conventional out 
siders view is to say that open source software is not owned. It is fear of the lack of 
accountability associated with this perceived lack of ownership that makes many com 
panies reluctant to deploy open source software for mission-critical functions. 

In fact, this conventional view is deeply mistaken. To understand why, we must 
make a distinction between "ownership" and "stewardship." Ownership is something 
that is fully transferable from one owner to another without loss of value. Money, 
and many (though not all) material objects, are examples of entities that can be sim 
ply owned. Stewardship, on the other hand, applies when something undergoes 
change, when it evolves, or when it has some kind of life cycle. In this case, some 
thing has value only if it is cared for in such a way as to sustain the life cycle. In an 
agrarian society, animals are a prime example of something requiring stewardship. 
Skills are required, and effort must be put forth, to maintain a herd. Transferring the 
herd to someone who lacks those skills or is unable to put forth the effort dimin 
ishes the value of the herd. In other words, only a good steward can realize the full 
value of that which is stewarded. 

Historically, land has been, and continues to be, at the center of contention between 
these two notions of ownership. For example, Native Americans considered them 
selves stewards of the land, and thus fell victim to the European notion of landown- 
ership. Today the battle between environmentalists and certain corporations is over 
exactly these two conflicting senses of ownership. Andrew Hessel s essay points to a 
brewing conflict in these senses of ownership with the biotech industry. 

In the technology sector, open source developers believe that software requires stew 
ardship. The standard employment contract and the proprietary software it engen 
ders preclude stewardship. Open source software, however, by its very nature 



encourages stewardship. Again, the motivation here is not altruism or charity. To an 
open source developer, stewarding software is the best way to see that the software 
evolves and improves, and hence it s just pragmatism to take a stance toward intel 
lectual property that assures that software will be stewarded. 

The proof is in the longevity of open source software projects and the stewards who 
tend them. Linus Torvalds is still at the head of the Linux kernel "tribe" more than a 
decade after the first public release of Linux. Eric Allman has guided Sendmail for more 
than 20 years. Larry Wall is still the guiding vision behind Perl, again after more than 
20 years. In these and many more cases, a common core group stood behind the soft 
ware for far longer than most proprietary software enjoys the benefits of a common 
development team. It is this the dynamics of stewardship far more than the "legions 
of programmers" that accounts for the success of open source software. 

Further, it is this dynamic of stewardship that fosters the social network around open 
source software that is based on the reputation game. Having committed themselves 
to what they regard as the most pragmatic approach to intellectual property, open 
source hackers have then adopted the professional and social structure needed to 
support that approach to intellectual property. It isn t altruism. It s chivalry, a far 
subtler and more pragmatic thing. 
* * * 

The Burning Man event lasts for one week each year. Contained within that event is 
the remarkable community of Black Rock City. Yet it would be a mistake to assume 
that the community exists for only one week a year. The complex structure and intri 
cate hierarchy that is the Burning Man community could not adapt, evolve, and sus 
tain itself if it were not a year-round phenomenon. 

That is the deeper truth not obvious to the casual observer: Burning Man is a perma 
nent worldwide community whose members are connected to and engaged with each 
other continuously. Art projects that are on exhibit at Burning Man will often be 
shown at smaller gatherings in places such as San Francisco. Burners gather for regu 
lar social events throughout the year to talk about past events and plan for next year. 
When Washoe County or the Bureau of Land Management considers a change that 
may affect the permit for Burning Man, word travels through the community like 
wildfire, and burners show up in force to make their cases and state their views. 

It s no accident that the growth of Burning Man parallels the growth of the Internet. 
The Burning Man web site is an impressive knowledge archive about the event and 
the community, providing a wealth of information and resources for anyone trying to 
understand Burning Man and learn how to get involved. The "Jack Rabbit Speaks" 
mailing list provides an announcement forum that goes out to the whole of the Burn 
ing Man community, but there are dozens of other mailing lists that tie together 
smaller communities within. Some of these are organized by geographical proximity, 



XXXVIH x> Introduction 



but many more are organized by common interest. A theme camp will often have a 
mailing list for its members. The Black Rock Rangers have their own web site, and 
their own mailing lists. 

What the Internet has done is to free us from the constraints of geography in terms of 
whom we connect to, whom we share common interests with, and whom we form 
community with. The power of intentional community is abundantly clear in the 
open source movement, where developers from around the world can collaborate on 
software of common interest. Yet the larger lesson is that the power of collaboration 
and the power of community exhibited in open source have relatively little to do 
with software development. 

We begin to see the lasting significance of open source only when we see that it is 
one instance of a general pattern of online, collaborative community. Even a very 
physical, tangible event like Burning Man is crucially dependent on this larger sense 
of community. If we look closely, we see that this pattern of collaboration is begin 
ning to manifest itself in many other places beyond open source. 

We see this strikingly in Eugene Kim s essay, contrasting an early example of soft 
ware collaboration with the grass-roots collaboration that emerged around the 
Ground Zero cleanup. We see this in the power of consumer-driven innovation that 
Sonali Shah explores. We see it in the sense of community behind Slashdot, 
described by Jeff Bates and Mark Stone, and the spontaneous movement that became 
Groklaw, described by Pamela Jones. And we see a compelling attempt to distill the 
most general patterns of these communities in Steve Weber s essay. 

The simplest elements are these: 

Recognizing that one has common cause with people who might otherwise have 
competing or divergent interests 

Acknowledging that small teams working on a component of a problem are the 
only scalable way to tackle large problems 

Improving solutions iteratively through a sense of stewardship, ecosystem, and 
evolution, rather than a sense of property and ownership 

Taken together, these principles suggest an organizational structure that is at once 
novel and familiar. These intentional communities form hierarchically, but it is a hier 
archy based on achievement and reputation rather than power, money, or authority. 
Communication flows easily up and down the hierarchy, but decision-making flows 
from the top down. "Everyone gets a voice, but not everyone gets a vote" (see Bates and 
Stone). The resulting organization is more tribal than democratic. 



XXXIX 



Each night when the two burners return to camp in the hours before dawn, "Reflec 
tions in Sand" has changed shape. Some new pattern has been carefully raked into 
the black-lit sand, and though they never see the visitors, the evidence of their pas 
sage and participation is there. 

The last evening is the night of the Burn. All day the Man has been laid down flat as 
he is prepped. At sunset he is raised up again. While all week his arms have been 
down at his side, now they are raised high above his head. This is the signal for the 
ceremony to begin. 

A ring of lights surrounds the Man, and the rangers walk the perimeter to ensure 
everyone keeps their distance. For hours, within the ring of lights, drummers, 
firedancers, and musicians perform. Pagan rituals from 1,000 years ago must not 
have looked so different. When at last the Man ignites, flames shooting 50 feet or 
more into the night sky, there is awed silence. Pieces begin to fall off, and he begins 
to tremble, as only guy wires hold him in place. The trembling increases, and at last 
the whole Man collapses into a burning mound. At that moment the crowd rushes 
the center in a wild, swirling dance that brings them as close to the flames as heat 
will permit. 

Now when the two burners return to their camp, they find the sand has been raked 
again, into one last new pattern. And something more; there, in the very center of the 
little Zen garden, someone has left a bottle of water. One of them reaches in to 
retrieve the bottle and pulls off the lid. 

"You going to drink that?" asks the other. 

"Of course. Out here, water is the most precious gift of all." 

An hour later their camp is packed. The black-light-painted rocks go back with 
them, the sand has been scattered, and the wooden frame and rakes have been 
heaped onto their neighborhood burn pile. They drive toward the gate, and the 
attendant waves them down. 

"Heading out?" asks the attendant. 

"Yeah, back to San Francisco," replies the driver. 

"This your first time here?" 

The passenger answers, "It is for me." 

The attendant gives a knowing smile and waves them through. "See you next year. 
Welcome to the tribe!" 



XX 
XX 



SECTION 1 



Open Source: 
Competition and 
Evolution 



In Section 1, we present essays tied directly to the history and development of open source soft 
ware. These essays can be loosely grouped into three categories: 

Essays on the software development process (Baker, DiBona, Allison, and Laurie) 

Essays on business competition and open source (Olson, Murdock, Asay, Walli, and 
Nelson) 

Essays on policy issues related to open source (Seltzer; Gonzalez-Barahona; 
Sharma and Adkins; Yeo, Liu, and Saxena; and Souza) 

The essays on the development process provide a natural extension from the original Open Sources. 
These essays explore the community and process that open source developers comprise, and explore 
the subtle similarities and differences between open source and proprietary development. 

With the original publication of Open Sources in 1999, the idea of an open source business model 
was something of a novelty. Today, we see in these essays, that open source, both in its licensing 
structure and in the commoditizing effect of its distribution model, has become a powerful tool in 
the hands of businesses large and small. 

One critical aspect of the business dynamics behind open source is the desire to avoid vendor 
lock-in through proprietary software, and to control one s own technology destiny. While these 
issues matter to businesses, they have become fundamental policy issues in Europe and develop 
ing nations. Control of technology resources in the coming decades will likely matter as much as 
control of natural resources has in the last century. Avoiding monopoly by a single company, or 
hegemony by a single nation, has become a paramount policy objective. Increasingly, open source 
is becoming the means of achieving that objective. 





b CHAPTER 1 

Mitchell Baker 




The Mozilla Project: Past and Future 



The Mozilla project was launched on March 31, 1998. On this date, the source code 
for the Netscape Communicator product was made publicly available under an open 
source license, the "Mozilla Organization" was founded to guide the project, and 
development of the codebase began to move from a proprietary model into an open 
model coupled with commercial involvement and management practices. 

Of these three elements, the release of the source code is discussed in Open Sources. 
In summary, the source code was prepared for public release by removing all code 
that Netscape didn t have the right to license under an open source license, and then 
replacing those pieces necessary for the code to compile and run. At the same time, a 
new open source license the Mozilla Public License was written, reviewed, and 
accepted by the open source community, including the Open Source Initiative (http:// 
www.opensource.org). The other two topics the story ofmozilla.org and the develop 
ment of the Mozilla project are the subject of this essay. The creation of the Mozilla 
Public License is generally an untold story, but it occurred during the time covered 
by the original Open Sources book and isn t discussed in detail here. 

Each of these three activities was a step into the unknown. Basic development princi 
pals of the open source model ("running code speaks," peer review, leadership based 
on technical merit) were known. But the combination of open source techniques 
with an active, focused commercial management structure was uncharted territory. 
The shift of authority from a commercial management structure to a separate organi 
zation was new, and presented many management challenges. The development of 



project management techniques and tools that could be shared by multiple commer 
cial development teams and a volunteer community was new. Development of a 
large, complex end-user application in the open source space was new. 

Of course, the Mozilla project was not the first open source project with commercial 
involvement. Cygnus, many of the Linux distributors, and Sendmail were all compa 
nies involved with open source development, and the Apache project was develop 
ing experience in coordinating open source development where some of the contrib 
utors were paid by their employers. But none of these projects provided more than a 
rough set of guidelines for how the Mozilla project might operate. The Mozilla 
project was unusual, and at the time perhaps unique, in the way project leadership 
interacted closely with both commercial teams (project managers, people managers, 
and engineers) and individual contributors. 

Not all open source projects are interested in commercial project management and 
people management issues, but for us it was always a given. Today other projects are 
thinking about these issues as the development and use of open source software 
increase. Given our history, size, and scope, the Mozilla project remains a trendsetter in 
this arena. 

Founding of the Mozilla Organization: Obvious for Developers, a 
Bold Step for Management 

The Mozilla project originally grew out of Netscape Communications Corporation 
and its Netscape Communicator product. In early 1998, the Netscape management 
team made the decision to continue development of Netscape s flagship product, 
Netscape Communicator, through an open source development model. At the time, 
Netscape Communicator and Microsoft s Internet Explorer browser were locked in a 
fierce competitive battle often referred to as the "browser wars." Netscape s goal was 
to seed a broad-based development effort within the software development commu 
nity to produce future browser products as a shared resource. 

At its inception, the Mozilla project faced some paradoxes. First, the only people 
familiar enough with the code to participate actively in its development were 
Netscape employees. Those employees were still expected to work within the man 
agement system and practices that Netscape had developed in its proprietary days. 
There was no volunteer community. And yet, even at that early time, it was clear that 
the long-term success of the project required a broad constituency of people and 
companies working jointly on the project. It was not enough to have open source 
code (code available under an open source license). The project needed an open 
development process, and this required authority over the code s development to be 
based on technical merit and distributed outside Netscape. The question was how to 
get there from here. 



* C The Mozilla Project: Past and Future 



One thing was clear: the success of the project depended on it being a real open 
source project. In other words, the project needed to have technical legitimacy and 
development decisions would need to be guided by technical considerations. This 
was intuitively clear to the group of Netscape employees who were familiar with 
open source, eager to help move the Mozilla code into the open source world and 
who ultimately became the founding members of the Mozilla Organization. This 
group made the need clear to Netscape management, which was receptive to trying 
to do the right thing. 

When the Mozilla project was officially launched, Netscape executive management 
therefore took some bold steps. First, they officially anointed "mozilla.org" as the 
steward of the codebase and leader of the project. I say officially because it s quite 
possible that a group like mozilla.org would have developed even if Netscape hadn t 
officially helped to create one. But this step was important, as it allowed mozilla.org 
to focus on building the project rather than on proving the necessity of its role. 

The creation of mozilla.org was a significant step that set the tenor of the project s 
development. It meant that the development of the Mozilla codebase was to be 
guided by something other than Netscape s own product and revenue plans, and also 
that Netscape management would need to give up control. This may seem like an obvi 
ous statement in an open source world, but it is one of the most difficult problems in 
moving from a proprietary to an open system. It is particularly difficult when the 
commercial vendor is actively trying to ship a product and the code has not yet 
reached a good, solid state that can serve as the basis of that product. 

Some have said that the Mozilla project was not a true open source project during this 
time because Netscape employees contributed so much to the project and Netscape 
management was so involved for so long. It s possible this is true. But I believe that 
Netscape management lived in an intensely uncomfortable setting as control of the 
project moved from their hands into those of mozilla.org. And since I personally was 
the fulcrum for stresses between the project leadership and the Netscape management 
team, I ll warrant that Netscape management felt it was living in a real open source 
project. In 1999, Netscape Communications Corp. was acquired by America Online 
(AOL). This resulted in many changes, but the relationship between mozilla.org and 
the Netscape browser development group continued as before. For quite a while, we 
used the term Netscape/AOL to describe the Netscape browser development group after 
the AOL acquisition, and I ll use that phrase for the rest of this chapter. 

The members of the Mozilla Organization are known as "mozilla.org staff." The original 
members of the 1998 launch were Netscape employees who had a vision for an open 
source Mozilla project and a determination to see it succeed. The most media-genie of 
these founders was Jamie Zawinski, who left the project after a year. But the most con 
sistent and long-term contributor has been Brendan Eich, who was a founding member 
of mozilla.org and remains the technical and philosophical leader of the project to this 



Founding of the Mozilla Organization: Obvious for Developers, a Bold Step for Management 



day. Over time, the percentage of mozilla.org staff employed by Netscape decreased 
steadily. Today, the mozilla.org staff does not have any Netscape/AOL employees. 

After the Mozilla project was launched, mozilla.org staff members began the process 
of changing development styles from a proprietary to an open source model. The 
early steps were logistical: establish public communications channels such as mail 
ing lists and newsgroups; establish a public system for viewing and tracking bugs. A 
harder task was changing habits. For example, the existence of public communica 
tions channels was not enough. Old habits die hard, and there was a tendency for 
people to use the methods they had always used. This was complicated by the fact 
that Netscape as a company still had confidential data about itself and its business 
partners that couldn t go into public forums. So, it was not possible to eliminate all 
private channels. Eventually we changed the names of any remaining private mailing 
lists to something long and awkward that required conscious thought to use. This 
gave Netscape/AOL employees a way to disseminate confidential data when neces 
sary, but made public disclosure the easiest path. 

Even as basic a step as public communications in an open source project can be diffi 
cult for some management teams to accept. In a system that is public by default, 
everyone needs to learn what information must remain confidential, and to remem 
ber this while working. At first it s a big effort to work in public and some people see 
it as overhead. Then as the project progresses and the public interaction provides 
increasing value, the need to keep something private is seen as a burden. This is cer 
tainly the case for the Mozilla project today, where the only private development 
information we solicit are bugs which could have an impact on the security features 
of our products. We ve set up a system for treating these bugs privately, and the sys 
tem has overhead. We bear it in the security context because security is critical, but 
we avoid it in other contexts. 

We also set up a public bug and issue-tracking system. This is commonplace today, but 
was innovative at the time. We made the bug-tracking system an open source project 
under the Mozilla umbrella, and today Bugzilla is a successful project in its own right 
(http://www.bugzilla.org). We also set up a continuous build system and web frontend 
(http://tinderbox. mozilla.org/showbuilds.cgi). This means that we have an automated pro 
cess that builds and rebuilds the software continuously on multiple platforms to see if 
and when a new piece of code causes the software not to build. Then came a period of 
learning to "work in the fishbowl." Some people adapt easily to having all their work vis 
ible, and others struggle. Many simply walk down the hallway to talk with a buddy, and 
then forget to tell everyone else. This period takes some time and effort. 

Updating the Codebase 

About six months into the project, it became clear that the codebase was in need of 
updating. By late 1998, the inherited code was several generations old, had been 
patched over and over, and actually hindered ongoing innovation. Old and fragile, it 



^ C The Mozilla Project: Past and Future 



looked backward toward the beginning of the Web, rather than forward to the new 
technologies a modern browser would need to support. So, in late 1998, a painful deci 
sion was made to rewrite the layout engine, a critical and complex core component. 

This decision changed the scope of the project dramatically. The initial project was 
an incremental upgrade from the Netscape Communicator 4.x product line to a pro 
posed 5.x product line. Moving to the new layout engine (known as Gecko) meant 
that the incremental model was gone; the Mozilla project would need to develop a 
complex new layout engine and then build a new browser application on top of it. 
And things got harder from there. As the new layout engine began to mature, it 
became clear that other significant parts of the codebase would also need rewriting. 
The development process turned out to be a lot like a remodeling project where fix 
ing one problem leads to another. Then came the long, slow grind to producing 
something useful (Mozilla 1.0 in June 2002) and finally something great (Mozilla 
Firefox in November 2004). 

During this time, many proclaimed us dead, a failure. What those people didn t see 
was the passion and commitment of the contributors to the project, including the 
Netscape employees. The contributors knew that they were developing good technol 
ogy. They knew they had a shot at building a great browser and mail client. And they 
knew it mattered. The Web matters. Browsers matter. Much of the world decided 
that the days of browser innovation were over. Some mourned the loss of choice, and 
many didn t realize the dangers of accessing the World Wide Web only through a 
single access point. But the contributors to the Mozilla project realized both the dan 
ger and the potential for something innovative. They persevered. The prominence of 
Netscape often obscured the efforts and dedication of the individuals themselves. 
Yes, many contributors were paid by Netscape. Of these, many contributed far 
beyond the requirements of a job, doing extra work to make the product "theirs" and 
to make sure they were proud of it. Meanwhile, the individual volunteers provided 
critical expertise and contributions across the codebase. 

During this period, almost all the code in the Mozilla browser and email client was 
rewritten. The focus was a modern layout engine, and a set of technologies designed to 
make the promise of cross-platform development a reality for the Web. We created a 
cross-platform component model (XPCOM),a cross-platform XML-based UI language 
known as XUL (pronounced zool\ a new toolkit using XUL, and a set of cross-platform 
applications themselves. Developing these was a long process, but we felt that it was 
important to have technology that would help us move forward. The power of these 
technologies has been demonstrated through our new products: Mozilla Firefox and 
Mozilla Thunderbird, in which we were able to build award-winning cross-platform 
applications quickly on top of mature, preexisting infrastructure. 



Founding of the Mozilla Organization: Obvious for Developers, a Bold Step for Management 



*.* 

XX 



A Disciplined Methodology 

Along the way, the Mozilla projects developed a highly disciplined method of distrib 
uted software development. Many people think that open source development is nec 
essarily chaotic. Or they wonder about the quality of the code because anyone can 
create a patch and offer it for inclusion in the source base. Open source need not be 
chaotic, and the Mozilla project is not. For every piece of code checked into the 
Mozilla products, we track: 

Who checked it in 

When it was checked in (to the minute) 

What problem it was trying to address 

The complete history of the issue (bug) the code was trying to address 

Who did the code review (often two levels of review) 

Whether the next build of the software was broken on any of our main platforms 

Whether the code affected our performance metrics, by platform 

Build and optional log comments 

A comparison with the previous version of the code. 

This information is available at any time; it does not require an expert to find or 
assemble the data. It is available online, in real time, through a web interface; all one 
needs is a web browser (http://tinderbox.moz.illa.org/showbuilds.cgi). We do this so that 
many contributors can work on the same codebase simultaneously and know what s 
going on. We do this so that the source code "tree" stays healthy, and we address 
problems before more new code makes them worse. We also have policies determin 
ing who gets access to the CVS tree, what s required before code can be checked in, 
what to do when the tree doesn t compile and run, how authority is delegated to 
those with expertise, and so on. 

Building an Open Source Project 

The process of building software proceeded simultaneously with building a vibrant 
open source project. The creation of mozilla.org had gone hand in hand with the rec 
ognition that Netscape management would need to give up a great deal of control over 
the development process. Now it was time to figure out how to make the transfer. I 
describe in this section two of the most significant topics we addressed control over 
the source code repository and control of the designated releases in some detail, for I 
believe this shows how the Mozilla project came of age. 

Implementing a transfer of control from Netscape management to mozilla.org caused 
a number of strains. Mozilla.org staff could have proceeded in opposition to 
Netscape. Indeed, we thought about it many times. However, Netscape was a large 



* * The Mozilla Project: Past and Future 



and valued contributor, whose involvement and work product remained very impor 
tant for the project. So, we spent a great deal of energy figuring out techniques that 
addressed the concerns a commercial entity like Netscape brings to a project and 
simultaneously building a strong open source community. 

Having control of the source code repository may seem like an obvious requirement 
for an open source project, but there were many sensitivities in our case. For exam 
ple, the Mozilla project has processes to make sure that code is of good quality before 
it is checked in. This is often the case in open source projects, and many contribu 
tors understand that their code needs to meet project standards. However, this is not 
always the case in commercial settings. Often the employment decision is what mat 
ters. If someone is an employee, his code goes into the project he was hired to work 
on. Instituting code review for everyone, even people employed to contribute code, 
can be a surprise for new employees and for their managers. Suddenly the decision of 
what code goes into the source repository is not made by managers for a particular 
person through the employment process. Instead, the decision is made by engineers 
who review the code itself rather than the person s credibility or employment status. 
And the code review is not optional or auxiliary; code cannot be checked into the 
tree until an appropriate reviewer has given formal approval. 

We instituted code review as a prerequisite to check-in relatively early in the process 
without too much controversy. Sometime later we implemented a second layer of review 
which we ended up calling "super-review." Super-review is an "integration review" or 
"plumbing review." Does the code conform to coding guidelines? Does it use the some 
what tricky XPCOM in appropriate places and not elsewhere? Does it conform to the 
overall architectural goals? Does it avoid needlessly diminishing modularity? 

We instituted the super-review requirement because we felt we had to. The Mozilla 
codebase is large and complex, and we were worried about overall code quality. We 
needed to increase our confidence that the code we were accepting into the tree would 
solve the immediate problem, and still leave us enough flexibility for future develop 
ment. Implementing the super-review was very painful for everyone involved. It was 
painful for the identified super-reviewers. Being a super-reviewer is actually a lousy 
job who wants to review yet more code instead of writing it oneself? And of course, 
the super-reviewers were well regarded and had plenty of work to do themselves. They 
took on the super-review job not because they wanted to, but because they believed it 
was important to making Mozilla the project we wanted it to be. Super-review was also 
painful for the contributing engineers. Adding another layer of review to the pre-check- 
in requirements was seen as "yet more bureaucracy" by some. 

There were many requests for absolute, guaranteed turnaround times for super- 
review. Engineers and managers complained they could not schedule work accu 
rately due to the unknown lags caused by waiting for super-review. And yet, the 
super-reviewers had to balance review of code with writing their own. And since they 

Founding of the Mozilla Organization: Obvious for Developers, a Bold Step for Management * J 3 



were well regarded, their code was acutely needed. I was unwilling to agree to rigid 
turnaround times for super-review, believing that doing so would put the work of 
key contributors at risk. We eventually agreed on a timeframe for an initial response 
from the super-reviewer, which would contain some estimated time for full review. 

Another issue was determining who has "write-access" to the source code repository. 
This was extremely sensitive. In a commercial setting, it is often the case that when 
people are hired they are given access to the part of the source code repository to 
which they are expected to contribute. In open source settings, one typically earns 
access by making valuable contributions. Moving the open source standard of "earn the 
right to touch the tree" to employees who need to contribute code to do their job can 
be difficult. In particular, the rationale for why this is necessary can be hard to explain 
in a commercial setting when all the code an employee writes must be reviewed and 
super-reviewed before it can be checked in. The question comes up: assume Employee 
X is hired. She writes code, and it passes review and super-review. Why on earth can 
that employee not go through the mechanical task of actually checking the code in? I 
was never able to provide a complete answer to this question. I know that open source 
projects regularly vet people before allowing access to the source code repository. And I 
know it would be very odd for a management chain in a company to make the deci 
sion about CVS access. So it s clear that this "is just not done." But I was not able to 
explain clearly how someone could do damage by having CVS access if all of her code 
was reviewed and approved before check-in, even though the technical leadership of 
the project felt very strongly about this. The idea of automatic access for employees had 
an emotional response because open source projects rely on peer review and technical 
leadership, and I shared this. But the key engineers were adamant that the quality of 
the code would suffer through automatic access, even though our pre-check-in code 
review requirements are quite stringent. 

It was an awkward setting to institute a policy for which I couldn t give a crisp reply 
to the various management teams affected. (I ve been responsible for policy for 
mozilla.org since 1999, so it was my job to write the policy, describe it to manage 
ment, and address the concerns and complaints that might come up.) And the possi 
bility existed that management groups would be surprised, distressed, or outraged 
that such a policy would be instituted without a clear answer as to why the code 
quality wasn t adequately protected. Nevertheless, we instituted this policy in 2001. 

A second area where shifting control to mozilla.org was highly sensitive concerned 
control of the milestone releases. By "owning the releases," I mean several things: 
first, defining and implementing a planned milestone schedule; second, defining and 
implementing a process for getting a release into shipping condition; and finally, 
identifying and shipping the release. 

When the Mozilla project was launched, the planning and release schedule was deter 
mined and implemented by Netscape employees contributing to the project, but not 



10 x* The Mozilla Project: Past and Future 



directly by mozilla.org staff. We worked on changing this for quite some time before 
we had proven ourselves trustworthy enough for Netscape to give up control. This may 
seem laughable how can a so-called "open source" project not control its own 
releases? But it s important to remember that Netscape was under enormous pressure 
to release a product, and giving up control of the process by which releases are made is 
extremely uncomfortable. Mozilla.org was new and unproven. And of course, manag 
ing a software project with hundreds of people working on it is not easy in any setting. 

Control of these aspects moved formally from the Netscape management to the 
mozilla.org staff following the release of the Netscape 6 browser. This shift had been 
discussed for some time but still involved a leap of faith for those who had previ 
ously exercised decision-making power. 

This was a tempestuous time; mozilla.org staff had influence, but not control. We 
thought many times about whether we needed to create a fork in order to affect such 
a shift in control. Each time, the mozilla.org staff decided that Netscape s contribu 
tions were far too important to the project and outweighed the desire for open source 
purity or credibility. I suspect that the Netscape management team must have had 
similar discussions, weighing a fork and the ability to manage their releases as they 
felt best with the value gained from the open source project. In the end, we all hung 
in and mozilla.org staff became the official keeper of our releases after Netscape 6. 

Owning our releases did not mean that we ignored Netscape. Netscape remained the 
largest single contributor, and its pool of talented and dedicated engineers was bog 
gling. Netscape was also the largest single distributor of Mozilla-based products. 
Managing the project without taking Netscape s needs into account would have been 
stupid. By this time, Netscape didn t exercise control as it once had, but its leader 
ship role in the project was greater than some might have liked. Mozilla.org staff con 
tinued to hear from the community that we were Netscape stooges and there wasn t a 
"real" open source project. However, to my mind, Netscape s role was now deter 
mined by classic open source principles: leadership and influence through the qual 
ity of one s contribution. Of course, this involves an acceptance of the role of a cor 
porate entity in an open source project, which made some uncomfortable. 

Mozilla.org was able to own our releases well, in part because we had developed an 
active, effective quality assurance (QA) community. Over the years, I ve learned how 
few people intuitively grasp the importance of the QA effort (and I suspect, how 
important QA will be to other projects of similar scope). Both of our major efforts 
web browsing and email live in an extremely complex world. The Web is very 
diverse, and people use our software in a boggling array of environments and in 
wildly different ways. Hiring a QA team as full-time testers is part of a solution, but it 
is not the complete answer. I m becoming more and more convinced that it may well 
be impossible to hire a QA team big enough and diverse enough to do thorough test 
ing of a product like a web browser. 

Founding of the Mozilla Organization: Obvious for Developers, a Bold Step for Management * 11 



By 1999, it had become apparent that an active community of people was interested in 
contributing to our testing and QA effort. Christine Beagle joined us to lead an effort at 
mozilla.org to make this group cohesive, figure out ways to give the group some author 
ity, and encourage them to step forward. The response to a bit of attention and apprecia 
tion directed at this nascent community was astounding. One mark of success is that 
shortly thereafter, we hired one of the more active and organized of these folks as our 
community QA lead. This person was Asa Dotzler, who has been a key figure in the 
project ever since, still coordinates QA activities, and is extremely active in managing 
our release process. With Asa s coordination, we began to see a set of people doing more 
organized testing of our products. This provided enormous value. The testers also did 
things such as look at all the bugs assigned to a particular engineer, verify that the bugs 
were legitimate, check to see if the bug existed on all platforms, create test cases, and 
then verify fixes across our main platforms. These efforts saved enormous amounts of 
time for the engineers trying to write the code to fix the bugs. I ve learned that this type 
of work often gets little respect, but we value it highly and find these contributors to be 
extremely important to our project. Many of these efforts were coordinated through 
mozillaZine, an independent webzine dedicated to the Mozilla project that was founded 
by Chris Nelson in September 1998 (http://www.mozillazine.org). 

Massive community testing remains important today. We provide Release Candi 
dates for our major product releases precisely so we can get 50,000-100,000 down 
loads from our key community and get a good reach on quality. 

Young Adulthood the Mozilla Foundation 

The idea of an independent legal organization to guide the Mozilla project had been 
discussed when the project was first launched in 1998. However, it was decided that 
the time was not quite right. At the time, there were no models for setting up such an 
organization and figuring out how it would be governed, who would participate, and 
so on. There was enough unknown and far too much work in getting the code ready, 
the project launched, and a browser developed to take on things we didn t abso 
lutely have to do. Eventually we decided that the right time to create an independent 
Mozilla Foundation would be when a critical mass of people was interested in sup 
porting a foundation. That critical mass would need to include a significant set of 
volunteers and a set of companies interested enough to fund browser developer and 
distribute Mozilla-based technology. 

That critical mass began to develop with the release of Mozilla 1.0. Mozilla 1.0 
showed that we could produce a good product, that the Mozilla releases where deter 
mined by Mozilla rather than by Netscape, and that the project had a positive future. 
At least one critical corporate participant came to us and told us that 1.0 proved our 
viability and that they were very interested in helping form and support an indepen 
dent Mozilla Foundation. 



12 X The Mozilla Project: Past and Future 



Following the release of Mozilla 1.0, I spent a fair amount of time thinking about 
what an independent Mozilla Foundation would look like, how we might put it 
together, how many employees we would need, which companies would likely pro 
vide support, and how to finance employees in the early years. I had help from a set 
of mozilla.org staff members. In addition, I had the good fortune of hooking up with 
Mitch Kapor, who had recently joined the open source world with the launch of the 
Open Source Applications Foundation (http://www.osa/oundation.org). Mitch was an 
immense help in thinking through various possible structures for the Mozilla Foun 
dation and is an unsung hero in getting the Mozilla Foundation launched. 

In the spring of 2003, the stars aligned. Mozilla.org staff was ready, the project had 
developed a critical mass, and we had some corporate support. In addtion, AOL decided 
it was ready to help spin out the Mozilla project. This was an important element for 
mozilla.org staff. Of course, we could have launched a project without AOL s support 
that s the nature of open source but the mozilla.org staff felt that AOL s support was 
important to the launch of an independent Mozilla project. We hoped that the use of the 
Mozilla trademarks would be transferred to a new organization, along with a set of 
machines. We wanted to be able to hire a group of people, some of whom were current 
AOL employees, without bad feelings. We felt it was very important to the project s sta 
bility to have a smooth transition from AOL to a successor. We also knew we needed to 
hire people to keep the project running well, and that it would take us time to find 
ongoing funding sources. So, the seed funding that AOL provided was another critical 
factor. Through July, I worked to reach agreement with AOL on how the Mozilla Foun 
dation would be launched. Once again, Mitch Kapor provided invaluable assistance in 
helping to get the arrangements with AOL worked out. 

On July 14, 2003, the Mozilla Foundation was launched as an independent non 
profit organization. AOL contributed $2 million in seed funding for the Mozilla 
trademarks, the Mozilla Public License, the machines we were using to host the web 
site and other infrastructure, and the efforts of a transition team to help create a 
smooth handoff. We knew we had some additional funding from IBM and Sun, and 
Mitch Kapor donated $150,000 for each of the first two years. Based on this, Bren 
dan Eich and I decided, with the help of Chris Hoffman and Mitch, to aim for an ini 
tial group of 10 employees. 

The initial group was divided among (i) those focused on the projectwide resources 
(technical leadership, infrastructure, tools, web site management, builds, releases, 
QA), (ii) those focused on the codebase itself (Firefox, Thunderbird, Gecko, the 
DOM, and JavaScript) and (iii) a couple of people focused on all the other things the 
project and the Mozilla Foundation needed to be successful, including relationships 
with commercial contributors and other organizations, legal structure, trademarks, 
finances, and so on. Mitch Kapor offered to extend his organization for providing 
back-office services payroll, benefits, accounting, donation processing, and human 



Young Adulthood the Mozilla Foundation * C 13 



resources to the Mozilla Foundation on very gracious terms, which has been a great 
boon. Securing high-quality services in these areas for the Mozilla Foundation had 
always been of concern to me and this has been a phenomenal solution for us. 

This resulted in a group that was small for the scope of the project, but still big for a 
nonprofit open source project to support. We chose this route because we believed that 
the project was unlikely to reach its potential without a core group of at least this size. 
We felt this was the minimum size for critical mass for several reasons, including these: 

The World Wide Web isn t finished. 

It changes all the time. New content types develop, new technologies develop, 
and new possibilities emerge. If the browser doesn t continue to develop, the 
consumer s ability to enjoy these enhancements stagnates. 

Browsers and email clients aren t done yet. 

There s a whole range of innovative ideas that interact with browsers and email 
clients. For example, RSS readers can be nicely integrated with both browsers 
and email. In addition, the underlying components on which the actual end-user 
applications are built require constant development. 

Speed matters. 

We need good Internet clients now. Having a core set of people able to devote 
full-time attention to this makes a big difference in accomplishing things quickly. 

The size and scope of the project requires it. 

Just keeping track of what s going on in the Mozilla project takes time. About 80 
people actively check into the CVS repository each month, and of course, many 
more active participants don t have CVS access. We also have a high level of 
involvement with commercial entities and with Mozilla development teams at com 
mercial entities. Providing the technical leadership and coordination for this large a 
group is a big job, even with a set of full-time employees. Doing so without a set of 
people available full time (or more than full time) would be beyond daunting. 

The founding in July was followed by a hectic startup period through the fall. We 
assembled the team of employees. We found office space at an affordable rate, thanks 
again to friends of the Mozilla project who extended a helping hand. We moved our 
equipment from AOL to our co-location facility and our offices. We knew it was 
important that enterprises and other institutions got a good picture of the Mozilla 
Foundation and grew confident that we are not a naive, flaky group, so we spent a 
chunk of time talking with these groups. 

We decided that a serious focus on the end user needed to be added to our traditional 
focus on developers. Product development continued at a fast clip through this period. 



14 X The Mozilla Project: Past and Future 



Firefox and Thunderbird 

As if forming the Foundation, moving employees, and establishing and supporting 
ourselves wasn t enough, we also began a determined transition from the application 
known as Mozilla or the Mozilla Application Suite, or by its codename, Seamonkey, 
to our new products: Mozilla Firefox and Mozilla Thunderbird. 

We knew that our future lay with the new applications. The integrated Mozilla 
Application Suite is a fine product that many love. But the integration caused diffi 
culties, the UI had been built by accretion and had been added to over the years, and 
we knew we wanted updated, standalone browsing and mail applications. Given our 
limited resources, we had to place a bet, and we did. 

The Mozilla Foundation hired the lead Firefox and Thunderbird developers, Ben 
Goodger and Scott McGregor. We talked with Ben and Scott about providing assis 
tance to the community of people working on the Mozilla Application Suite. We con 
tinued with our releases of the Mozilla Application Suite, including improvements to 
the core components, performance, stability, and security, and coordinating feature 
work done by our community. But we did not hire people focused on the Mozilla 
Application Suite. 

Both Firefox and Thunderbird were in the early stages of development when we 
made this decision. Indeed, Thunderbird had not even seen its 0.1 release when the 
Foundation was launched. Despite this, we knew that the then-current state of Thun 
derbird could probably have supported an 0.1 or 0.2, or maybe even an 0.3 label. 
This was borne out when we were contacted over the summer by a Fortune 100 
company wanting information about Thunderbird. The company had already done a 
significant amount of due diligence and had realized that Thunderbird was the best 
option. They wondered if the Foundation would be interested in speeding develop 
ment of certain enterprise features if we had some additional funding to do so. As a 
result, Thunderbird has had a rich set of enterprise features from its early days. It 
lacks an integrated calendar, but the Mozilla calendar project was reinvigorated and 
an integrated calendar project launched in the fall of 2004. 

Firefox was further along the development path, but still quite young. It wasn t even 
called Firefox at that time; it was called Firebird, the second of two early names 
which we abandoned due to trademark issues. The application-eventually-known-as- 
Firefox was at the 0.5 stage, quite usable, but not a polished end-user application. 
The development goal had always been a strict focus on the end-user experience 
above all else. This continued, and Bart Decrem drove the end-user focus through 
out all aspects of the Foundation s operations. Firefox began to be noticed in 2004 
with the 0.6 release. It quickly began to capture the interest of much of our devel 
oper community. There were still millions of contented users of the Mozilla Applica 
tion Suite, but the momentum had clearly begun shifting to Firefox. 



Young Adulthood the Mozilla Foundation * J 15 



In February 2004, we found a public posting by a visual designer named Steve Gar- 
rity, describing what Firefox needed for its icons, logos, and visual identity in gen 
eral. The content of the post was excellent. Better yet, Steve seemed to have both 
knowledge of and an interest in tackling these problems, instead of simply complain 
ing or pointing out problems. We asked him if he d like to take the lead for a bit and 
show us what could be done. He said yes, and the Visual Identity Team was created. 
Both Firefox and Thunderbird took a giant step toward becoming sophisticated, pol 
ished end-user applications. By the 0.8 release of Firefox in June 2004, the momen 
tum for Firefox was growing dramatically. Firefox was already an impressive prod 
uct, offering features new to most users. 

In addition, the Internet experience had become extremely painful. Malicious actors 
were everywhere. The Web was infested with viruses and security exploits, seemingly 
uncontrollable pop-up windows appeared almost everywhere, and distracting, band 
width-chewing ads appeared long before desired web content. The browser, a piece of 
software many had come to take for granted, suddenly mattered. The browser is the 
mechanism through which one s computer one s hard drive with its critical and pri 
vate data-connects to the wild, wild world of the Web. A modem, high-quality 
browser is necessary to keep this connection from being increasingly painful and even 
dangerous. The Mozilla Foundation had a great browser in Mozilla Firefox, and people 
began to notice. By mid-2004 we began to see that the types of people who were inter 
ested in Mozilla Firefox were expanding. We began getting messages from people who 
clearly were neither early adopters nor even particularly savvy. So, we knew we were 
making a difference. And we knew that the difference was important enough for more 
people than ever before to pay attention. People who tned Firefox loved it. Around the 
0.9 timeframe Qune 2004), a groundswell began building. 

The summer of 2004 was an even more painful time on the Web. A series of viruses 
and security issues caused enormous inconvenience and concern. Internet Explorer was 
a vector for many attacks. These problems caused consumers to pay more attention to 
their browser. They helped people realize why an alternative browser is so important to 
the health of the Internet and one s ability to interact comfortably with the Web. Secu 
rity is a very difficult problem. A browser must be open to the content of the Web 

that s the whole point. At the same time, it can t be too open. A browser needs to have a 
series of defenses to help filter out bad content. No browser can be perfect, and that 
includes Firefox. We know that we will be making security changes and improvements 
in our products on a continual basis, and we hope that others do as well. 

We saw significant adoption of Firefox 0.9 through the summer and fall of 2004 
almost 8 million people came to get a product that hadn t reached its 1.0 status yet. 
Mozilla Thunderbird adoption was also proceeding well, though not at the same fan 
tastic rates. On the marketing side, Spread Firefox was launched in September 2004. 
This was a community marketing effort, perhaps the first of its kind. We knew that 



IB * * The Mozilla Project: Past and Future 



the great strength of the Mozilla project is the community of people dedicated to 
making it successful. We also knew that we would not have a traditional "market 
ing" or "PR" effort, spending large amounts of money on media events. And the mail 
we were receiving made it clear that people were excited about Firefox and wanted to 
help their friends and family switch. 

The result was www.spreadfirefox.com, the home of a fervent evangelism community 
focused on increasing Firefox adoption. The most famous Spread Firefox campaign 
to date has been the New York Times campaign, which was proposed and initiated by 
a community member. This started out as a 10-day campaign to get 2,500 people to 
contribute funds to buy a full page ad in the New York Times supporting Firefox. 
Ten days was the wrong timeframe 2,500 people signed up in the first two days. 
We kept the campaign open for 10 days anyway and ended up with 10,000 choos 
ing to participate. We had promised that contributors names would be in the ad and 
would be legible, so we enlarged the ad and made it a full two-page ad. It ran on 
December 16, 2004. A while later I came to work to find two young men standing 
outside our door. The door is glass and we had taped the NYT ad to the door so that 
it was visible from the outside. The two young men looked lost, but one wore a Fire- 
fox T-shirt. So, as I reached the door to go in, I asked, "Can I help you find some 
thing?" The men were rather shy, looking at their feet and mumbling, "We just 
wanted to see the Mozilla Foundation. We re only in town for a few days and had to 
see it." Then one of them straightened up, looked me in the eye, jabbed his finger at 
the New York Times ad, and said proudly, "And there s my name, right there!" Some 
times I think people believe I m exaggerating when I describe how passionate con 
sumers are after they ve tried Firefox, but it s actually hard to overstate the excite 
ment that Firefox has generated. 

Getting Firefox and Thunderbird to a 1.0 status and shipped was a very intense 
period. We knew we had great products in the works, but we had to finish them. We 
also had to get a set of related activities completed. These included revamping our 
web site, developing our communications plan, working with the Spread Firefox 
community, improving our localization process and working with the various local 
ization communities, figuring out our search relationships, working with our affili 
ates Mozilla Europe and Mozilla Japan on the international aspects of the launch, 
and so on. The ferocious dedication of everyone involved was required. I cannot 
stress enough the commitment of the Mozilla community. On Sunday, November 7, 
I logged onto IRC at about 8:00 A.M.., which is early for me and for most of the 
Mountain View-based staff. I was bombarded with questions from our localization 
communities in Europe and Asia. Some were up early, many were up very, very late, 
and all were trying to figure out how to manage their schedules over the next 48 
hours to be available whenever needed to get their localized versions finished, 
approved and shipped as part of the 1.0 release. 



Young Adulthood the Mozilla Foundation 



Mozilla Firefox 1.0 and Mozilla Thunderbird 1.0 were released on November 9, 2004. 
To say they have been well received is an understatement. Firefox 1.0 was downloaded 
from our mirror site about 2 million times in the first two days alone, and has plunged 
on at an average rate of almost 250,000 downloads per day since then. As of mid-April 
2005, the number of downloads that we can track is very close to 50 million. On the 
usage side, Firefox has gained worldwide market share at a rate of nearly 1% per 
month from November to April. As of April 2005, surveys are beginning to show 
Mozilla browsers at or above 10% market share. Among technically focused sites, the 
market share of Mozilla products ranges up to much higher numbers. 

It s extremely difficult to gain this sort of market share on the desktop in the face of a 
competitive product that people get when they buy a computer. The fact that Mozilla 
Firefox has done so is a reflection of a great product, a huge need, a fervent commu 
nity, and the power of the Internet. 

Many people have wondered whether open source development can produce great 
end-user applications. One school of thought says that open source developers can 
produce infrastructure and products that other developers like, but not applications 
aimed at the general end user. Mozilla Firefox and Thunderbird demonstrate that 
open source software can indeed produce great end-user products. I believe that we 
are only at the beginning, and we will see a range of innovative end-user products 
come from the open source world in the coming years. 

The Future 

The mission of the Mozilla project is to promote choice and innovation on the Web 
by creating great end-user offerings. We focus on innovation because the Web is still 
young we ve seen only the beginnings of its potential. That potential can be stifled 
if we don t have innovative work done on the client side. 

We focus on choice because this allows people to have greater control over their 
Internet experience. This control over our life on the Web increases in importance 
each year, as more and more critical functions such as banking, health care, insur 
ance, and commerce are done over the Web. A monoculture is rarely a healthy ecol 
ogy. A single effective choice in browsers and email clients is dangerous, both to con 
sumers and to the health of the Web itself. 

Firefox in particular has shown that consumers will pay attention to a product that 
provides an alternative, and that the Mozilla project can create such a product. We 
have a number of challenges ahead of us. We need to continue to release products 
that people love. We have a set of responsibilities that come with the user base, 
adoption rate, and increased visibility of the project. Conditions will change, and we 
will need to adapt. These are challenges, but certainly no greater than those we have 
faced to date. These are the challenges that result from the project s achievements. 



18 X The Mozilla Project: Past and Future 



We have great talent, a powerful and creative community, a well-earned place in the 
Internet ecosystem, a growing user base, and, at long last, a legal home for the 
Mozilla project in the Mozilla Foundation. 

As we go forward, there is no change in the mission of the project. Our basic 
approach of combining open source DNA with involvement by commercial entities 
will continue. The Mozilla Foundation has grown some and may grow some more, 
and we expect to continue working closely with a set of companies that are inter 
ested in developing and distributing Mozilla technology. The increasing acceptance 
of open source software by the commercial world opens up greater possibilities for 
collaboration. The emergence of web-based services provided through the browser 
also encourages business models for the service provider other than charging for each 
copy of software provided. This allows more entities to contribute to our project. 
Our focus on distributed development, technical excellence, and welcoming new 
participants will continue. The need for a vibrant, creative community of people 
focused on the Web will not change. 

I expect the Mozilla project will continue to be a trendsetter in a number of arenas: 
development of open source end-user products, combining volunteer and commer 
cial activity in an open source project, maintaining a critical mass of people as 
employees of the Mozilla Foundation, and funding that set of employees plus com 
munity marketing and adoption programs. We aren t the only ones doing these 
things, and we continue to learn and benefit enormously from the open source 
projects. We hope to contribute ever more in return. 



19 





b CHAPTER 2 

Chris DiBona 




Open Source and Proprietary 
Software Development 



In this chapter, I present a perspective on the similarities, differences, and interac 
tions between open source and proprietary software development. 

Proprietary Versus Open Source? 

Before you go any further, throw off any notion that the proprietary developer is some 
how a different person from the open source developer. It is uncommon for a member 
of the open source developer community to do only open source for a living. Only the 
most prominent, or loaded, members of the open source community come close to 
having this kind of freedom. It is indeed rare to find a developer who develops only 
with proprietary tools and libraries. Even Visual C++ and C# developers benefit from a 
great variety of code and libraries that are free for use in their programs. 1 

My career has focused on open source development for the last 10 years, and I m 
constantly pleasantly surprised by how open source development and proprietary 
resemble each other. I believe this is because proprietary developers are educated by 
the adventures of their slightly crazy open source cousins, but I also know that open 
source developers have learned just as much from proprietary developers. 



1 Traditionally, one difference between open source and proprietary development teams has been 
that open source teams are, in general, geographically quite dispersed. However, in this age of 
outsourced, offshored, and distributed development, even proprietary development has 
become highly dispersed geographically. 



Don t read this as an attempt to muddy the difference between proprietary and open 
source programs. They are different, sometimes very much so. However, they come 
from the same people, and they re using a lot of the same methods and tools. It is the 
licenses and the ideals behind open source programs that make them remarkable, 
different, and revolutionary. 

The Example Culture 

A lot of people, when talking about open source software development, say that open 
source developers enjoy a great productivity gain from code reuse. This is true, but 
in my experience all developers, not just open source developers, benefit from the 
existence of free-of-charge standard libraries and code snippets. For decades, propri 
etary developers have had a great variety of prepackaged libraries to choose from, but 
these proprietary libraries haven t taken root in the same way that freely usable, open 
libraries have. 2 

Code reuse? Knowledge reuse! 

In Linus Torvalds essay from the first Open Sources, he talked about how the rise of 
open code was delivering on the promise of reuse touted by proponents of the Java 
programming language specifically and object-oriented programming in general. 

That said, it has been my experience that there is a point at which software develop 
ers will go out of their way to avoid reusing code from other projects. In some shops, 
they call it "not invented here" (NIH) syndrome, and some companies are famous for 
it. But even those shops use standard kernels, libraries, and compilers. The real diffi 
culty here is in figuring out where the NIH line lies. Although the answer is different 
for every single programmer and team, all still can (and still do) learn from the open 
code out there, which is a unique advantage of open code. While both open and pro 
prietary code can be reused in a wide range of circumstances, open code enables 
something further: knowledge reuse. By examining the code itself, the developer can 
learn how a particular problem is solved, and often how that solution is an instance 
of a general solution type. It is this kind of reuse that Linus applauded and that the 
NIH developer misses. 

Then why not simply use other people s code? There are a number of factors to con 
sider before code is incorporated, and these must be understood before one can 
understand the role that Free Software has had in development. 



This will likely inspire many to cite their favorite commercial library. A full survey of libraries, 
both commercial and open source, would be required to validate this statement properly. This 
is an educated assumption on my part, as when commercial libraries manage to gain any son of 
prominence, open source developers tend to fill the gap, thus overshadowing the commercial 
project. 



22 * C Open Source and Proprietary Software Development 




Speed of development 

There are very real barriers to using other people s code. You have to examine how to 
interface with said code, and you need to review the code to make sure it meets your 
standards for security, license, style, and correctness. You also need to integrate it 
into your version control and build system. 

None of these problems is insurmountable, but they have to be worth surmounting. 
To wit: if all I need is a routine to do something simple, such as iterate through an 
array of numbers and perform some simple operation on them, using someone else s 
software would be a waste of time. 

When developing, I like to use large libraries only when I either don t want to deal 
with a technology, or I don t fully understand it and don t feel qualified to imple 
ment it. For a recent project, I was pulling newsfeeds from weblogs and performing a 
kind of natural English language processing on it. I thought that using a tool called a 
"stemmer" to normalize the data would make my later analysis more accurate. 

Implementing the routines to download and process feeds could have taken a month 
or two, and this is exactly the kind of development I don t like to do. To properly 
implement a stemmer, I d likely have to get my graduate degree and then write it 
which would impact my deadline a bit so I downloaded programmer-friendly 
libraries that did each of these tasks. The stemmer was available under the Berkeley 
Software License, and the feed parser was available under the Python Software 
License, both of which are very easy to deal with and do not require any onerous 
post-incorporation duties. I was thus able to save time and have better code. 

That said, some things I m very interested in developing myself. Since I was doing 
this project as an excuse to learn a natural-language processing algorithm, which was 
interesting to me, I wanted to write that part of the program myself. I was (and am) 
also fascinated with a problem I think I ll have in storing the results such that I can 
quickly retrieve them from a database. I haven t solved that problem as of this writ 
ing, but I don t necessarily want to use other people s code for that. I have read some 
code and examples in textbooks and online that will help me with the former, but 
the storage problem is mine, for now. 

This gives you an idea where the line was for me in this particular project, but oth 
ers have the same reticence for other, subtler reasons. 

A particularly difficult codebase 

What makes software difficult to add to your code? Sometimes the code is simply in 
the wrong language. Maybe you are using Perl and want to tie some code into a C or 
Python module. That s not always so easy. Maybe the code was really developed on 
only one platform say, an Intel machine and you want it to work on your iBook, 
which runs on a PowerPC processor. 



Proprietary Versus Open Source? X 23 



The problems with using other people s code can be legion. Maybe their routines 
were implemented assuming a machine with a lot of memory or processor cache, 
making it perform poorly or, worse, unpredictably, 3 on your target platform. Maybe 
the software was developed for an earlier version of your programming language, so 
a lot of features you would have implemented with a standard library call are instead 
implemented from scratch, thus reducing future maintainability. 

Problems arise with canned libraries as they get older. For instance, the aforemen 
tioned feed parser library is useful because its author, Mark Pilgrim, is very good at 
keeping it up to date with the 13 "standards" that lie behind that "xml" button on 
your favorite blog or web site. If the library were to fall into disuse, or Mark were to 
stop working on it and no one else picked up the work, I d likely change to a differ 
ent library or choose to maintain it myself. 

There is another reason to not use someone else s code, and it will look amazingly 
petty to all but the programmers reading this. 

Technically speaking, this: 

int myfunction(int a) 

{ 

printf("My Function %d\n",a); 

} 
is the same as this: 

int myfunction(int a) { 

printf("My Function %d\n",a); 

} 

which is the same as this: 

int myfunction(int a){ printf("My Function %d\n",a);} 
and this: 

int myfunction(int a){ 

printf("My Function %d\n",a); 
} 

They compile to the same result on any given compiler. 

I could go on, but I won t. The point is that, depending on the programmer or dic 
tated company style, each of these is wrong, evil, bad, or awful, or perhaps one is 
acceptable. Not all programmers and companies care about style, but many (one 
might argue the smartest) do. The ones that do care actively dislike the ones that 
don t and do not want to use their code. Should they have to touch the offending 



3 This might seem strange, but programmers are OK with the odd performance hit sometimes. 
Unpredictable results lead to crashed programs, however. This is not good, no matter what 
you ve been told. 

24 * * Open Source and Proprietary Software Development 



library, they will inevitably have to make it "readable." Whether you call this refac- 
toring or prettifying or whatever, it can drive a programmer away from a hunk of 
code, unless it really brings something fantastic along with it. 

"My Goodness," you might consider asking, "are programmers delicate, petty crea 
tures?" No, there are some very good reasons to have consistent code style. It aids in 
debugging. Some say it reduces bugs (I d agree). It makes code navigation much faster 
and makes it easier for people to write tools to generate and manipulate code than they 
might otherwise. There are other reasons too, but I don t want to get too arcane. Some 
languages, such as Python, have very rigid appearance rules, as appearance can dictate 
how a variable can be used. Style may appear to be a trivial concern, but it isn t. 

Comfort 

Maybe you just want to do it yourself. Businesspeople in the industry who have grown 
up around open source often comment that duplication of effort, or "reinventing the 
wheel," is not time well spent. I rarely hear this from programmers. When people hear 
about KDE and GNOME, or Linux and BSD, or even more esoteric arguments about 
which window manager to use, inevitably someone will chime in, "Obviously, they had 
a lot of time on their hands. Otherwise, why would they have started from scratch?" 

The implication is that the programmers have somehow wasted time. When I choose 
to reimplement some technology or program, I know what I m doing, and even if it 
is a "waste" of time or duplication of effort, I think of it as practice. And when I can 
enjoy the luxury of implementing from scratch, I really like the results, because 
they re all mine and what I ve developed works exactly the way I want it to. 

But Why So Many of the Same Things? 

Business, of course, is interested in productive developers, and productive developers 
don t rewrite things, right? No, not necessarily. People rewrite code all the time. The 
more-informed companies recognize that this type of thing is often inevitable, and the 
best and most resourceful encourage this kind of mental knife sharpening, because it 
leads to better developers and better code. Given the time, programmers often prefer to 
leam from other people s code without actually using the code, and if open source ends 
up as one big repository of example code, I call that a success. 

Also, computers change. Computers, languages, compilers, and operating systems 
change so quickly that a periodic rewrite of some code becomes vital, from a perfor 
mance perspective. To take advantage of the newest processors, architectures, and 
other advancements, a recompile will certainly be required and will likely expose 
issues with your code (architecture changes lead to this directly). 4 



For example, you write a program on your handy laptop, you compile it, and it runs great. Later, 
you run it on your fabulous dual Opteron server. It crashes because you assumed an integer was 32 
bits and the Opteron (running a 64-bit OS like Linux, of course) has 64-bit integers. This is a basic 
error that comes up in a lot of different ways during 32-64 bit transitions. 



25 



But people are using libraries, code, and examples from open source code, copying 
them into their codebases rapidly. Certainly this happens. Don t let my counter cases 
fool you. It is a rare codebase that doesn t involve some open source software, 
whether it is merely in the form of a standard library or a widget library, or is full of 
the stuff. This is by design; if every program had to write every instruction down to 
the operating system, or the machine itself, there would be no programs. The itera 
tive building process, programs on top of libraries on top of the operating system, is 
so productive that I can t imagine someone ignoring it. Even for the smallest embed 
ded systems, designers are using the GNU compilers to create great programs for 
their devices: compile, flash, and go. 

Libraries, System Calls, and Widgets 

Here we begin to see how open source ideals have changed proprietary development. 
When proprietary software developers create a program, they may use free software, 
created or derived libraries, widget libraries, and tools. This includes developers target 
ing proprietary operating systems such as Windows and OS X. Developers creating 
software, whether for OS X, Unix/Linux, or Windows, commonly use free tools to do 
so. They almost always use free libraries in the creation of their programs and often use 
free user interface elements during the creation of their systems. 

Some might think I m indulging in some mission creep for free software here, 
assigning a larger role to it than it maybe should enjoy. I m not. I ll take it even fur 
ther: if there hadn t been free tools like the GNU compiler collection, the industry 
would have been forced to create and release them. Otherwise, the computer 
industry as we know it would not exist and would certainly not be as large as it is 
right now. This is not to imply that companies somehow owe something to the free 
software community. However, companies do help out when they can reap a long- 
term benefit. IBM understands this, as do Novell, Google (my employer), and 
many others. Even Microsoft uses and releases code under a variety of licenses, 
including the GPL (its Unix services for Windows) and BSD (Wix), but Microsoft is 
conflicted both internally and externally, so it s not as easy for it to embrace open 
source. 

Am I saying that without free tools, the compiler would try to charge a per-pro- 
gram fee? No, I think that if free tools hadn t arrived and commoditized the com 
piler, other competitive concerns would have kept the price of software develop 
ment tools accessible and cheap. That said, I think free tools played a big part. Free 
and open source software changed expectations. Microsoft and Intel make no 
attempts to prevent developers from using their compilers to create free software or 
software that is counter to their corporate goals. Client licenses, a common fixture 
in the email/workflow market, are unheard of for mainstream development tools. 



2B X Open Source and Proprietary Software Development 



If there is one thing about free software that is downright scary to proprietary devel 
opment shops, it may be this: software that is licensed per client almost always 
comes under attack from free software. This is forcing in the software industry a shift 
away from such per-client licenses in all but the most specialized verticals for 
instance, the software that runs an MR1 machine, or air traffic control software, both 
of which are so specialized as to not count, because every client is custom. The grand 
irony here is that in some industries, such a high cost is attached to developing soft 
ware that some are forming very open source-looking consortiums to solve common 
software development problems. 

Distributed Development 

Distributed development is more than just a fad or even a trend. Organizations and 
companies large and small are using diverse, globally distributed teams to develop 
their software. The free software development movement showed the world how to 
develop internationally. Well before SourceForge.net became a site that every pro 
grammer had heard of, projects working together over the Internet or far-flung con 
nected corporate networks developed much of the software that we use today. 

In fact, the tools they developed to do that are now considered the baseline standard for 
developerd everywhere. What company in its right mind doesn t mandate that its pro 
grammers use some form of version control and bug tracking? I ask this rhetorically, 
but for a long time in the software business, you couldn t make this assumption. Small 
development shops would back up their data, for sure, but that s not version control. 

Distributed development is about more than just version control. It s also about com 
munications and bug tracking and distribution of the end result of software. 

Understanding Version Control 

Programming is an inherently incremental process. Code, then build, then test. 
Repeat. Do not fold, spindle, or mutilate. 5 Each step requires the developer to save 
the program and run it through a compiler or interpreter. After enough of these 
cycles, the program can do a new thing or an old thing better, and the developer 
checks the code into a repository, preferably not on his machine. Then the reposi 
tory can be backed up or saved on a hierarchical storage system. Then, should a 
developer s workstation crash, the worst case is that the only work lost is that done 
since the last check-in. 



This sentence is famous for being printed on punch cards, an early way of providing computers 
with data. If they were folded, spindled, or mutilated, they jammed the readers which makes 
one speculate what the punch card programmer used for version control. The answer is right 
there in front of you: as the cards went through revisions, they swapped out cards and retained 
the old, original cards. 



Distributed Development * 27 



What is actually stored from check-in to check-in is the difference from one version to 
the next. Consider a 100-line program, in which three lines in a program read: 

for (i=i; i < l; i++) { 

printf("Hello WorldVn"); 
} 

and one link needs to be changed to: 

for (i=l; i < 100; i++) { 

printf( "Hello to a vast collection of worlds !\n"); 
} 

which would then be checked back in. The system would note that only one line had 
changed and store only the difference between the two files. This way, we avoid 
wasting storage on what is mostly the same data. 

The value of having these iterations can t be overstated. Having a previously known, 
working (or even broken) copy can help in the event of an editing problem, or when 
you re trying to track down a bug that simply wasn t there a revision ago. In desper 
ate cases, you can revert to a previous version and start from there. This is like the 
undo option in your favorite word processor, but one that persists from day to day. 

Version control isn t used just in development. I know of IT shops and people who 
keep entire configuration directories (/etc) in version control to protect against editing 
typos and to help with the rapid setup of new systems. Some people like to keep their 
home directory in a version control system for the ultimate in document protection. 
There is even a wiki project that sits on top of the Subversion version control system. 

Additionally, good version control systems allow for branching say, for a develop 
ment and a release branch. The most popular version control system that many open 
source projects use is CVS. 

CVS 

CVS, the concurrent versioning system, allows developers all over the world to work 
on a local copy of a codebase, going through the familiar "code, build, test" cycles, 
and check in the differences. CVS is the old standby of version control, much in the 
same way RCS/SCCS was before it. There are clients for every development environ 
ment and it is a rare professional developer who hasn t been exposed to it. 

Since it is easy to use and install and it enjoys wide vendor support, CVS continues 
to be used all over the world and is the dominant version control platform. 

Subversion 

Only the rise of Subversion has brought real competition to the free version control 
space. With a much more advanced data store than CVS and with clients available for 
all platforms, Subversion (SVN) is also very good at dealing with binary data and 



28 X Open Source and Proprietary Software Development 



branching. Both are things that CVS isn t very good at. SVN is also very efficient to use 
remotely, and CVS is not; CVS was designed for local users and remote use was tacked 
on later. Additionally, SVN supports a variety of access control methods, supporting 
any authentication scheme that Apache does (Subversion is an Apache project), which 
includes LDAP, SMB, or any the developers wish to roll for themselves. 

What About SourceSafe? 

SourceSafe isn t really version control. Local version control, whether CVS or Source 
Safe, is just backup, requiring a level of hardware reliability that simply doesn t exist 
on a desktop. Since SourceSafe is not designed to be used remotely, you take the life 
of your codebase in your hands when you use it. There are some SourceSafe remot- 
ing programs out there if you must use SourceSafe, but I can t recommend them so 
long as decent, free SVN and CVS plug-ins exist for Visual Studio. 

The Special Case of BitKeeper 

BitKeeper, which was written by Larry McVoy, was chosen by Linus Torvalds to use 
for version control for the Linux kernel. For the Linux kernel, BitKeeper was a very 
good choice, given the kinds of problems that arise with Linux kernel development. 
Written for distributed development, BitKeeper is very good at managing multiple 
repositories and multiple incoming patch streams. 

Why is this important? With most version control systems, all your repositories are 
slaves of one master and resolving differences between different slaves and masters 
can be very difficult. 

The only "problem" with the kernel team s use of BitKeeper was that BitKeeper was 
not a free software program, although it was available for the use of free software 
developers at no charge. I say was because Larry McVoy recently decided to pull the 
free version, thus making it impossible for Linux kernel developers to work on the 
program without paying a large fee. 6 

A great number of developers lamented the use of a proprietary tool for free software 
development, and the movement off BitKeeper, while disruptive, is a welcome change. 

BitKeeper is a tool designed with the open source software model in mind. It has 
found success among large proprietary development houses specifically because the 
problems that faced the kernel team in 2001 are the same ones that increasingly face 
proprietary development shops. All of these teams, not just those working on open 
source development projects, now face multiple, far-flung teams that are engaged in 
collaborative development and struggle to do it effectively. 



6 The kernel team is in the process of moving off of BitKeeper as of this writing. 



Distributed Development *C 23 



Collaborative Development 

You have a developer in Tokyo, a team in Bangalore, a team in Zurich, and a shop in 
Seattle, all working on the same codebase. How can you possibly keep the develop 
ment train from coming off the rails? Communication! 

IRC/IM/Email 

One might imagine that only now, with the advent of IM and VoIP, can developers 
keep up with each other. In fact, developers have stayed in touch in something 
approximating real time since the early days of Unix, when they began to have a 
great variety of communications tools to use. 7 Early on, two developers on the same 
machine used the Write or Talk Unix programs, which allowed for a simple 
exchange of text between users. This grew into Internet Relay Chat (IRC) and then 
Instant Messenger (IM). 

Email itself plays the most important role in development. It is the base packet of 
persistent knowledge that distributed developer teams have. Wikis are also taking 
hold as repositories of information. 

VoIP 

Strangely (to nondevelopers) voice simply hasn t caught on as a terrific tool for ongoing 
developer communications. While a regular conference call is useful for keeping every 
one moving in the same direction, the idea of vocal input while developing would drive 
many coders away screaming. The phone isn t evil, but maintaining an uninterruptible 
flow can be very important to developer productivity. Phones also do not create a logfile 
or other transcript that can be referred to later. Don t take my experiences for gospel 
here. Read the book Peopleware 8 for more information about this. Everywhere I ve ever 
worked, the one constant has been developers wearing headphones, but listening to 
music, not other developers yammering in their ears. 

SourceForge 

The online site SourceForge.net is the largest concentration of open source projects 
and code on the planet. SourceForge boasts some 100,000 projects and 1 million 
registered developers, and people use its integrated version control, project web 
hosting, file release mechanism, bug control, and mailing lists to write a vast amount 
of software. Pulling together these features on a free platform for open source devel 
opers proved to be a revolutionary concept. Before, people were left implementing 
this themselves with Bugzilla (a bug-tracking mechanism) and CVS or some other 
version control/bug-tracking facilities. 



7 In fact, the Unix "write" command allowed hackers in the 1970s to communicate in a fashion 
not so different from IM. 

8 Tom Demarco and Timothy Lister, Peopleware (New York, NY: Dorset House Publishing 
Company, 1999). 



30 * * Open Source and Proprietary Software Development 



SourceForge represents, for a lot of people, the next stage in developer environ 
ments. VA Software, the company that runs SourceForge through its Open Source 
Technology Group (OSTG) subsidiary, sells this sort of solution into the enterprise, 
as does the Brisbane, California-based Collab.net. 

Software Distribution 

While free software developers know how to code, what about getting the code in 
front of the user? In the early days of the Free Software Foundation (FSF), the answer 
was to send out tapes and disks to users who wanted the tools, for a reasonable fee. 
Now that so many people have connections to the Internet, boxed software is begin 
ning to show its age, but software producers are really just now learning from open 
source how to distribute software in this way. 

Dependencies 

When you compile a piece of software, you sometimes end up relying on libraries that 
you must call from your program to do some task. If you try to run the program with 
out the expected complement of libraries, it cannot run or it may run poorly. Open 
source developers have created some very smart packaging and installation systems and 
filesystem methods that can make this a more tractable problem. Once they created 
these packaging systems and combined them with the Internet, they got online updat 
ing. The irony is that, in a lot of ways, Linux and Unix were schooled in this by Win 
dows. A common complaint regarding Linux when comparing it to Windows and OS X 
is that software can be very difficult to install. One could argue that Windows isn t all 
that easy to install either, but since Windows is preinstalled on most computers, this is 
an argument that often falls on deaf ears. 

I don t think Linux developers have learned to do installation well yet. There are some 
standouts, but for the most part, installation ease is still a work in progress. One thing 
free and proprietary share is the appreciation for and development of online updating 
systems. This is something Linux distributions get very right. In short, once Linux is 
installed on your machine, it can be very easy to keep it up to date. 

Online Updating/Installation 

Online updating is a terrific way of getting software onto your machines. More 
importantly, it is a terrific way to maintain a secure system over time. Since Linux 
distributions don t have to worry about software license ownership, it is very easy for 
the software to determine whether to download a patch or fix, and thus many Linux 
distributions have systems to facilitate this. Proprietary software development houses 
such as Microsoft are still trying to figure this out. It is a hard problem when you mix 
it with licensing concerns. Additionally, when it s done wrong, you can literally crash 
thousands, or in the case of Microsoft and Apple, millions of machines, so it is really 
critical to do well. That the Debian and Fedora Core Linux distributions do this at all 
is quite a feat. 

Software Distribution X 31 



Want a sticky issue? Do you trust your software vendor to allow it to automatically 
update your software? For some, this question is heard in these ways: 

Do you trust Microsoft to update your operating system? 

Do you trust a bunch of bearded Unix programmers to update your system? 

How you react to these questions has a lot to do with the realities of how difficult the 
problem is, how successful previous auto-updates have been in the past, and how 
trusting you are which brings up the subject of the next section. 

How Proprietary Software Development Has Changed Open Source 

Open source isn t magic, and developers aren t magicians. No developer is immune 
to security problems and bugs creeping into his code. 

Bugs/Security 

Free, open, proprietary, closed. . ..Bugs happen. I think open source means fewer bugs, 
and people have written tens of thousands of words explaining how they agree or dis 
agree with me. One thing I know I m right about is that both kinds of code have bugs. 
Bugs persist longer in closed codebases, and their closed nature keeps bugs persistent. 

If I may paraphrase Socrates, "An unexamined codebase is dead," and by dead I mean 
killed by the hostile environment that is viruses, worms, crackers, and Trojans. Like 
bugs, security flaws happen in both free and closed software. As a project matures, it 
must assemble a mantle of testing and quality assurance (QA) techniques that are 
vital to its ongoing health. I think open source development has learned much from 
the processes that proprietary software development houses have come up with to 
support their paying customers. 

Testing and QA 

As projects mature, so do the testing suites around them. This is a truism for free and 
for closed software codebases, but the research around this originated in commercial 
software/hardware and in academia, and open source software has been a ready con 
sumer of this information. The most popular talk I attended lately was in unit testing 
for Python at the O Reilly Open Source Conference. The room was packed, with peo 
ple sitting in the aisles. Testing is huge and is required for any project, free or not. 

Project Scaling 

Scaling is hard. Whether we re talking about development group size, bandwidth, 
space, or whatever, scaling any programming project is nontrivial. 

Software development has its limits. Product teams can t grow too fast or too large 
without one of two things happening either disintermediating technology or 
project ossification. Fred Brooks s seminal book, The Mythical Man-Month, covered 



32 * * Open Source and Proprietary Software Development 



this in depth, and the existence of F/OSS development methodologies doesn t change 
that. In fact, the tools and changes free software has brought to prominence are all 
around disintermediation and disconnected collaboration. 

F/OSS isn t magic. It isn t breaking the speed of light. Most projects, with some nota 
ble exceptions, are composed of small teams, with one to three people doing the vast 
majority of the coding. If you care about project size, you would be well served by 
reading the findings of the Boston Consulting Group s study of open source software 
developers on SourceForge. This revealing study analyzes project metrics and moti 
vations. For one thing, you see that projects almost always comprise fewer than five 
active developers. Many projects have only one developer. 

So, what am I talking about when I say disintermediating technology? Look at it this 
way. Imagine that one person decides to create a cake from scratch. He d have to 
start with a cake mix and some milk and an egg, right? No! He d need chocolate, 
milk, flour, yeast, water, and the other ingredients, right? No! Just for the milk, he d 
need a cow, some food and water for the cow, a bench, a milk bottle, a chiller, a pas 
teurizer, a cap and a rag, some bag balm, and so on, right? Well, you re getting 
closer. The point is that we accept interfaces all the time, and the successful project 
finds these interfaces, formalizes them, and spreads the work out along these lines. 

We accept power at 120 volts at 60 hertz alternating current. We don t generate the 
power ourselves. We accept that we don t need to dig for oil, refine it, and pour the 
refined gas into our cars. We use interfaces with different systems all the time. Programs, 
too, have interfaces, and the success of a program is in how it manages these interfaces. 

Proprietary or not, a successful program is one that interfaces effectively between sys 
tems and teams working on these systems. Microsoft doesn t have 5,000 engineers 
working on Windows. It has them working on the kernel, the printing subsystem, 
the windowing system, the voice synthesis module, and other components. More 
importantly, it has groups that work on interfacing between the systems so that they 
(theoretically) work as a whole. Likewise for the Linux kernel; Linus interacts with a 
number of captains who control different subsystems, including networking, disk 
drives, memory, CPU support, and so on. Fractionation, when possible, is key, and 
when not possible, disastrous which is why groups working to integrate the whole 
and making sure the interfaces are appropriate can make all the difference in the suc 
cess or failure of a project. 

This interface management is something that free software has done very well. Many 
commercial developers would be well served to learn from open source s interface 
management practices. 



How Proprietary Software Development Has Changed Open Source 2 C 33 



Control 

Control is something customers and end users have never had over their code. You 
don t buy proprietary software, you rent it, and that rental can be rescinded at any time. 
If you read the end-user license agreements (EULAs) that accompany proprietary soft 
ware, you may be left with the feeling that you are not trusted and not liked all that 
much. For instance, in Microsoft Word s EULA, there is this charming note: 

You may not copy or post any templates available through Internet-based services 
on any network computer or broadcast it in any media. 

So, if you were to take a standard Microsoft Word template (which all templates are 
derived from) and make one that is suited to your business as, say, a publisher, you 
would be in violation of your EULA with Microsoft, and thus vulnerable to its lawfirm. 

Controlling your software destiny is something I consider extremely important. Take, 
for instance, my employer, Google. We are able to fix and change the Linux kernel to 
fit our very specific needs. Do we have to check with Linus or one of his lieutenants 
before, during, or after we change the network stack? No. If we were running NT on 
our machines, we would be unable to get such changes made, and were we to enter 
into a deal where Microsoft would incorporate our requested changes, we would in 
effect be informing a competitor of our development strategy. 

Another example is a recent service pack from Microsoft, which featured a firewall and 
antivirus package. This package, which is turned on by default after service pack installa 
tion, was aimed at stopping the viruses and Trojans endemic to the Windows experi 
ence. Funnily enough, it considered iTunes a virus and presented a fairly confusing mes 
sage asking the user to authorize the program s use of the network. 9 That Microsoft s own 
media player, which has common network access methods, wasn t impeded is telling. 

Your computer is not your own; you only borrow that which makes it useful, and 
when that is taken away, you are left with nothing but a toxic pile of heavy metals 
and aluminum. 

I think this is a subtle but important part of open source s popularity. Many people 
and companies are interested in controlling their own destiny, and Linux and other 
open source programs make this possible. 

Intellectual Property 

Free-software developers believe in intellectual property, probably more so than peo 
ple who never consider open source software. Developers creating open source have 
to believe, as the entire structure of the GPL, BSD, MPL, and other licenses depends 
on the existence of copyright to enforce the clever requirements of those licenses. 



iTunes has a nifty sharing mechanism whereby users stream music to other iTunes users over 
the network. It s pretty neat. 



34 * * Open Source and Proprietary Software Development 



When you hear people criticizing free-software developers as guiltless communists or 
pie-in-the-sky dreamers, it is worth remembering that without copyright, there can t 
be free software. 

Discussions concerning intellectual property and free software usually revolve 
around two issues: patents and trademark. Software can be patented, and things can 
be trademarked. Exactly how these intersect with free software is complex. Can a 
piece of software which is patented be released under the GPL and still hold to the 
letter of the license? Can a program name be trademarked and then released under 
the BSD and still be a meaningful release? Legal opinion and precedence thus far pro 
vide no definite answer. 

Open source developers are learning, though, paying attention to the current events 
around intellectual property and how it affects them. 

The reality of intellectual property is something modern developers are almost 
required to learn. Learning the laws concerning software is the way to protect them 
selves from those who might send the feds out to arrest them when they come to the 
United States. I know that sounds like I m typing this with tinfoil on my head, but I 
am not kidding. 10 

The problem with this learning process is that it does take time away from coding, 
which is not good and is a net loss for free software which may indeed have been 
the whole point. 

Some Final Words 

While open source software is about freedom and licenses, it is nonetheless true that 
open source costs less, under many circumstances, than proprietary software. This is 
an important aspect of free software. Additionally, it has to be cost competitive 
against other free products, just as software that costs money must compete against 
an open source/free offering. 



10 I wish I were, but I m not. It happened to Russian developer Dmitry Sklyarov, who intended to 
discuss his reverse engineering of the Adobe PDF file format at the DefCon developers 
conference in Las Vegas. Upon landing at McCarran International Airport, he was met by the 
FBI, which placed him under arrest under the auspices of the Digital Millenium Copyright Act 
on behalf of Adobe Corporation. As a result, Linux kernel developers no longer have a 
substantial meeting in the United States, choosing instead to meet in Canada and Australia, two 
countries that do not have similar laws and rarely extradite for intellectual property-related 
crimes of this nature. Developers felt this was necessary because the Linux kernel uses code that 
was reverse engineered. Reverse engineering, by the way, is what made Dell, Phoenix, AMI, 
AMD, EMC, and a large number of other companies both possible and profitable. 



Some Final Words ** 35 



Free Things Are Still Cheaper Than Expensive Things 

When I say "competes against other free products," I m talking about pirated copies 
of Windows, Office, SQL Server, Oracle, and many others competing against Linux, 
OpenOffice, MySQL, Postgres, and other best-of-breed free software applications. 
These applications are doing very well in environments that have little regard, legally 
or culturally, for software licenses. 

Free things have a velocity all their own, and people forget that. I ll leave you with a 
little anecdote from when I was working for a large law firm in Washington, DC. I 
was still in college studying computer science, and I ran the law firm s email net 
work during the day. This was 1996 or so, and TCP/IP was clearly the big winner in 
the network format wars versus NetBIOS and SNA, to a degree that no one could 
have appreciated. I was in the elevator with one of the intellectual property attorneys 
at the firm a fairly technical guy when he said something like: "You know, if TCP/ 
IP had been properly protected and patented, we could have rigged it so that every 
packet cost money; they really missed the boat on that one." 

Where would the Internet be if this was true? I don t know, but I do know one thing: 
the Internet would not be running TCP/IP. So, enjoy the freedom of open source 
software. It is there for you! 



36 X Open Source and Proprietary Software Development 





t ^4 CHAPTER 3 
Jeremy Allison 



A Tale of Two Standards 



It was the best of protocols, it was the worst of protocols, it was the age of 
monopoly, it was the age of Free Software, it was the epoch of openness, it was 
the epoch of proprietary lock-in, it was the season of GNU, it was the season of 
Microsoft, it was the spring of Linux, it was the winter of Windows.... 

Samba is commonly used as the "glue" between the separate worlds of Unix and 
Windows, and because of that, Samba developers have to intimately understand the 
design and implementation decisions made in both systems. It is no surprise that 
Samba is considered one of the most difficult Free Software projects to understand 
and to join, outclassed in complexity only by the voodoo black art of Linux kernel 
development. Samba really isn t that hard, however, once you look at the different 
standards implemented in the two systems (although some of the decisions in Win 
dows can cause raised eyebrows). 

In developing Samba, we re creating a bridge between the most popular standards 
currently deployed in the computing world: the Unix/Linux standard of POSIX and 
the Microsoft-developed de facto standard of Win32. In this chapter, I will examine 
these two standards from an application programmer s perspective. In doing so, I 
thought it might be instructive to look at the reasons why each of them exists, what 
the intention for creating the particular standard might have been, and how well they 
have stood the test of time and the needs of programmers. A historical perspective is 
very important, as we look to the future and decide what standards we should 



encourage governments and businesses to support, and what effect this will have on 
the software landscape in the early 21st century. 

Standard: (noun) A flag, banner, or ensign, especially. An emblem or flag of an 
army, raised on a pole to indicate the rallying point in battle. 1 

The POSIX Standard 

POSIX was named (like many things in the Unix software world) by Richard Stallman. It 
stands for Portable Operating System Interface-X, meaning a portable definition of a 
Unix-like operating system API. The reason for the existence of the POSIX standard is 
interesting and lies in the history of the Unix family of operating systems. 

As is commonly known, Unix was created in 1969 at AT&T Bell Labs by Ken 
Thompson and Dennis Richie. Not originally designed for commercialization, the 
source code was shipped to universities around the world, most notably Berkeley in 
California. One of the world s first truly portable operating systems, Unix soon splin 
tered into many different versions as people modified the source code to meet their 
own requirements. Once companies like Sun Microsystems and the original, preliti- 
gious SCO (Santa Cruz Organization) began to commercialize Unix, the original 
Unix system call API remained the core of the Unix system, but each company added 
proprietary extensions to differentiate their own version of Unix. Thus began the first 
of the "Unix wars" (I m a veteran, but I don t get disability benefits for the scars they 
caused). For independent software vendors (ISVs), such proprietary variants were a 
nightmare. You couldn t assume that code that ran correctly on one Unix would even 
compile on another. 

During the late 1980s, in an attempt to create a common API for all Unix systems, 
and fix this problem, the POSIX set of standards was born. Because no one trusted 
any of the Unix vendors, the Institute of Electrical and Electronics Engineers (IEEE) 
shepherded the standards process and created the 1003 series of standards, known 
as POSIX. The POSIX standards cover much more than the operating system APIs, 
going into detail on system commands, shell scripting, and many other parts of what 
it means to be a Unix system. I m only going to discuss the programming API stan 
dard part of POSIX here because, as a programmer, that s really the only part of it I 
care about on a day-to-day basis. 

Few people have actually seen an official POSIX standard document, as the IEEE 
charges money for copies. Back before the Web became really popular, I bought one 
just to take a look at the real thing. It wasn t cheap (a few hundred dollars, as I 
recall). Amusingly enough, I don t think Linus Torvalds ever read or referred to it 
when he was creating Linux; he used other vendors references to it and manpage 
descriptions of what POSIX calls were supposed to do. 



1 http://www.dictionary.com. 



Reading the POSIX standard document, however, is very interesting. It reads like a 
legal document; every line of every section is numbered so that it can be referred to 
in other parts of the text. It s detailed. Really detailed. The reason for such detail is 
that it was designed to be a complete specification of how a Unix system has to 
behave when called from an application program. The secret is that it was meant to 
allow someone reading the specification to completely reimplement their own ver 
sion of a Unix operating system starting from scratch, with nothing more than the 
POSIX spec. The goal is that if someone writes an application that conforms to the 
POSIX specification, the resulting application can be compiled with no changes on 
any system that is POSIX compliant. There is even a POSIX conformance suite, 
which allows a system passing the tests to be officially branded a POSIX-compliant 
system. This was created to reduce costs in government and business procurement 
procedures. The idea was that you specified "POSIX compliant" in your software pur 
chasing requests, the cheapest system that had the branding could be selected, and it 
would satisfy the system requirement. 

This ended up being less useful than it sounds, given that Microsoft Windows NT 
has been branded POSIX compliant and generic Linux has not. 

Sounds wonderful, right? Unfortunately, reality intruded its ugly head somewhere 
along the way. Vendors didn t want to give up their proprietary advantages, so each 
pushed to get its particular implementation of a feature into POSIX. As all vendors 
don t have implementations of all parts of the standard, this means that many of the 
features in POSIX are optional usually just the one you need for your application. 
How can you tell if an implementation of POSIX has the feature you need? If you re 
lucky, you can test for it at compile time. 

The GNU project suffered from these "optional features" more than most proprietary 
software vendors because the GNU software is intended to be portable across as many 
systems as possible. To make their software portable across all the weird and wonder 
ful POSIX variants, the wonderful suite of programs known as GNU autoconf was cre 
ated. The GNU autoconf system allows you to test to see whether a feature exists or 
works correctly before you even compile the code, thus allowing an application pro 
grammer to degrade missing functionality gracefully (i.e., not fail at runtime). 

Unfortunately, not all features can be tested this way, as sometimes a standard can 
give too much flexibility, thus causing massive runtime headaches. One of the most 
instructive examples is in the pathconf ( ) call. The function prototype for 
pathconf ( ) looks like this : 

long pathconf (char *path, int name); 

Here, char *path is a pathname on the system and int name is a defined constant giv 
ing a configuration option you want to query. The constants causing problems are: 

_PC_NAME_MAX 
_PC_PATH_MAX 

The POSIX Standard X 39 



_PC_NAME_MAX queries for the maximum number of characters that can be used in a file 
name in a particular directory (specified by char *path) on the system. _PC_PATH_MAX 
queries for the maximum number of characters that can be used in a relative path from 
the particular directory. This seems fine until you consider how Unix filesystems are 
structured and put together. A typical Unix filesystem looks like Figure 3-1. 




/usr /home | 

r~ ~~i r~ ~n 

bin I Jeremy | chris 




Figure 3-1. Typical Unix filesystem 

Any of the directory nodes, such as /usr/bin or /mnt, could be a different filesystem 
type, not the standard Unix filesystem (maybe even network mounted). In Figure 3-1, 
the /mnt/msdos_dir path has been mounted from a partition containing an old MS- 
DOS-style FAT filesystem type. The maximum directory entry length on such a system 
is the old DOS 8.3 maximum of 1 1 characters. But below the Windows directory could 
be mounted a different filesystem type with different maximum name restrictions 
maybe an NFS mount from a different machine, for example, on the path /mnt/msdos_ 
dir/nfs_dir. Now the pathconf ( ) can accommodate these restrictions and tell your 
application about it if you remember to call it on every single possible path and path 
component your application might use! Hands up, all application programmers who actu 
ally do this.... Yes, I thought so. (You at the back, put your hand down. I know how 
you do things in the U.S. Star Wars missile defense program, but no one programs in 
ADA anymore, plus your tests never work, OK?) This is an example of something that 
looks good on paper but in practical terms almost no one would use in an actual appli 
cation. I know we don t in Samba, not even in the "rewritten from scratch with correct 
ness in mind" Samba4 implementation. 

Now let s look at an example of where POSIX gets it spectacularly wrong, and why 
this happens. 

First Implementation Past the Post 

Any application program dealing with multiple access to files has to deal with file 
locking. File locking has several potential strategies, ranging from the "lock this file 
for my exclusive use" method, to the "lock these 4 bytes at offset 23 as I m going to 
be reading from them soon" level of granularity. POSIX implements this kind of 



functionality via the fcntl( ) call, a sort of jack-of-all-trades for manipulating files 
(hence "fcntl - file control"). It s not important to know exactly how to program 
this call. Suffice it to say that a code fragment to set up such a byte range lock looks 
something like this: 

int fd = open("/path/to/file", 0_RDWR); 
Now, set up the struct flock structure to describe the kind of byte range lock we need: 

int ret = fcntl(fd, F_SETLKW, &flock_struct); 

If ret is zero, we got the lock. Looks simple, right? The byte range lock we got on the 
region of the file is advisory. This means that other processes can ignore it and are 
not restricted in terms of reading or writing the byte range covered by the region 
(that s a difference from the Win32 way of doing things, in which locks are manda 
tory; if a lock is in place on a region, no other process can write to that region, even 
if it doesn t test for locks). An existing lock can be detected by another process doing 
its own fcntl ( ) call, asking to lock its own region of interest. Another useful feature 
is that once the file descriptor open on the file (int fd in the previous example) is 
closed, the lock is silently removed. This is perfectly acceptable and a rational way of 
specifying a file locking primitive; just what you d want. 

However, modern Unix processes are not single threaded. They commonly consist of 
a collection of separate threads of execution, separately scheduled by the kernel. 
Because the lock primitive has a per-process scope, this means that if separate 
threads in the same process ask for a lock over the same area, it won t conflict. In 
addition, because the number of lock requests by a single process over the same 
region is not recorded (according to the spec), you can lock the region 10 times, but 
you need to unlock it only once. This is sometimes what you want, but not always: 
consider a library routine that needs to access a region of a file but doesn t know if 
the calling processes have the file open. Even if an open file descriptor is passed into 
the library, the library code can t take any locks. It can never know if it is safe to 
unlock again without race conditions. 

This is an example of a POSIX interface not being future proofed against modern 
techniques such as threading. A simple amendment to the original primitive allow 
ing a user-defined "locking context" (like a process ID) to be entered in the struct 
flock structure used to define the lock would have fixed this problem, along with 
extra flags allowing the number of locks per context to be recorded if needed. 

But it gets worse. Consider the following code: 

int second_fd; 

int ret; 

struct flock lock; 

int fd = open( "/path/to/file", 0_RDWR); 



First Implementation Past the Post X 41 



/* Set up the "struct flock" structure to describe the 
kind of byte range lock we need. */ 

lock.ljtype = F_WRLCK; 
lock.l_whence = SEEK_SET; 
lock.l_start = 0; 
lock.l_len = 4; 
lock.l_pid = getpid( ); 

ret = fcntl(fd, F_SETLKw, &lock); 

/* Assume we got the lock above (ie. ret == o). */ 

/* Get a second file descriptor open on 
the original file. Assume this succeeds. */ 

second_fd = dup(fd); 

/* Now immediately close it again. */ 

ret = close(second_fd); 

What do you think the effect of this code on the lock created on the first file descrip 
tor should be (so long as the close ( ) call returns zero)? If you think it should be 
silently removed when the second file descriptor is closed, congratulations you 
have the same warped mind as the people who implemented the POSIX spec. Yes, 
that s correct. Any successful close ( ) call on any file descriptor referencing a file 
with locks will drop all the locks on that file, even if they were obtained on another, 
still-open file descriptor. 

Let me be clear: this behavior is never what you want. Even experienced programmers 
are surprised by this behavior, because it makes no sense. After I ve described this to 
Linux kernel hackers their responsse have been that of stunned silence, followed by 
"but why would it do that"? 2 

The reason is historical and in my opinion, reflects a flaw in the POSIX standards pro 
cess, one that hopefully won t be repeated in the future. By talking to longtime BSD 
hacker and POSIX standards committee member, Kirk McKusick (he of the BSD dae 
mon artwork), I finally tracked down why this insane behavior was standardized by the 
POSIX committee. As he recalls, AT&T took the current behavior to the standards 
committee as a proposal for byte range locking, as this was how their current code 



To discover if this functionality was actually correctly used by any application program or if 
anything really depended on it, Andrew Tridgell, the original author of Samba, once hacked the 
kernel on his Linux laptop to write a kernel debug message if ever this condition occurred. After 
a week of continuous use, he found one message logged. When he investigated, it turned out to 
be a bug in the exportf s NFS file exporting command, whereby a library routine was opening 
and closing the /etc /exports file that had been opened and locked by the main exportf s 
code. Obviously, the authors didn t expect it to do that either. 



42 x> A Tale of Two Standards 



implementation worked. The committee asked other ISVs if this was how locking 
should be done. The ISVs who cared about byte range locking were the large (at the 
time) database vendors, such as Oracle, Sybase, and Informix. All these companies did 
byte range locking within their own applications, and none of them depended on, or 
needed, the underlying operating system to provide locking services for them. So their 
unanimous answer was "we don t care." In the absence of any strong negative feedback 
on a proposal, the committee added it "as is" and took as the desired behavior the spe 
cifics of the first implementation, the brain-dead one from AT&T. 

The "first implementation past the post" style of standardization has saddled POSIX 
systems with one of the most broken locking implementations in computing history. 
My hope is that eventually Linux will provide a sane superset of this functionality 
that can be adopted by other Unixes and eventually find its way back into POSIX. 

OK, having dumped on POSIX enough, let s look at one of the things that POSIX 
really got right and that is an example worth following in the future. 

Future Proofing 

One of the great successes of POSIX is the ease in which it has adapted to the change 
from 32-bit to 64-bit computing. Many POSIX applications were able to move to a 64- 
bit environment with very little or no change, and the reason for that is abstract types. 

In contrast to the Win32 API (which even has a bit-size dependency in its very 
name), all of the POSIX interfaces are defined in terms of abstract datatypes. A file 
size in POSIX isn t described as a "32-bit integer" or even as a C-language type of 
unsigned int, but as the type off_t. What is off_t? The answer depends completely 
on the system implementation. On small or older systems, it is usually defined as a 
signed 32-bit integer (it s used as a seek position so that it can have a negative value), 
and on newer systems (Linux, for example) it s defined as a signed 64-bit integer. As 
long as applications are careful to cast integer types only to the correct off_t type 
and use these for file-size manipulation, the same application will work on both 
small and large POSIX systems. 

This wasn t done all at once, because most commercial Unix vendors have to provide 
binary compatibility to older applications running on newer systems, so POSIX had to 
cope with both 32-bit file-sized applications running alongside newer 64-bit-capable 
applications on the new 64-bit systems. The way to make this work was determined by 
the Large File Support working group, which finished its work during the mid-1990s. 

The transition to 64 bits was seen as a three-stage process. Stage one was the original 
old 32-bit applications; stage two was seen as a transitional stage, where new ver 
sions of the POSIX interfaces were introduced to allow newer applications to explic 
itly select 64-bit sizes, and stage three was where all the original POSIX interfaces 
default to 64-bit clean. 



Future Proofing 2* 43 



As is usual in POSIX, the selection of what features to support was made available 
using compile-time macro definitions that could be selected by the application 
writer. The macros used were: 

_LARGEFILE_SOURCE 

_LARGEFILE64_SOURCE 

_FILE_OFFSET_BITS 

If _LARGEFILE_SOURCE is defined, a few extra functions are made available to applica 
tions to fix the problems in some older interfaces, but the default file access is still 32 
bit. This corresponds to stage one, described earlier. 

If _LARGEFILE64_SOURCE is defined, a whole new set of interfaces is available to POSIX 
applications that can be explicitly selected for 64-bit file access. These interfaces 
explicitly allow 64-bit file access and have 64 coded into their names. So, open( ) 
becomes open64( ), lseek( ) becomes Iseek64( ), and a new abstract datatype called 
of f 64_t is created and used instead of the of f _t file-size datatype in such structures 
as struct stat64. This corresponds to stage two. 

_FILE_OFFSET_BITS represents stage three; this macro can be undefined or set to the val 
ues 32 or 64. If undefined or set to 32, it corresponds to stage one (_LARGEFILE_SOURCE). 
If set to 64, all the original interfaces such as open( ) and lseek( ) are transparently 
mapped to the 64-bit clean interfaces. This is the end stage of porting to 64 bits, where 
the underlying system is inherently 64 bit, and nothing special needs to be done to make 
an application 64-bit aware. On a native 64-bit system that has no older 32-bit binary 
support, this becomes the default. 

As you can see, if a 32-bit POSIX application had no embedded dependencies on file 
size, simply adding the compile-time flag -D_FILE_OFFSET_BITS=64 would allow a 
transparent port to a 64-bit system. There are few such applications, though, and 
Samba was not one of them. We had to go through the stage-two pain of using 64-bit 
interfaces explicitly (which we did around 1998) before we could track down all the 
bugs associated with moving to 64 bits. But we didn t have to rewrite completely, 
and I consider that a success of the underlying standard. 

This is an example of how the POSIX standard was farsighted enough to define some 
interfaces that were so portable and clean that they could survive a transition of 
underlying native CPU word length. Few other standards can make that claim. 

Wither POSIX? 

The POSIX standard has not been static; it has managed to evolve (although some 
would argue too slowly) over time. A major step forward was the establishment of 
the Single Unix Specification (SUS), which is a superset of POSIX developed in 1998 
and adopted by all the major Unix vendors and shepherded by the Unix standards 
body, "the Open Group." It was a great leap forward when this specification was 

44 xC A Tale of Two Standards 



finally made available for free on the Web from the Open Group web site at http:// 
www.unix.org. It certainly saved me from having to hunt down cheap POSIX specifi 
cations in secondhand bookshops in Mountain View, California. 

The expanded SUS now covers such issues as real-time programming, concurrent pro 
gramming via the POSIX thread (pthread) interfaces, and internationalization and 
localization, but unfortunately it does not cover file Access Control Lists (ACLs). Sadly, 
that specification was never fully agreed on, and so has never made it into the official 
documents. Interestingly enough, the SUS also doesn t cover the GUI elements, 
because the history of Unix as primarily a server operating system has meant that GUIs 
have never been given the priority necessary for Unix to become a desktop system. 

Looking at what happened with ACLs is instructive when considering the future of 
POSIX and the SUS. Because ACLs were sorely needed in real-world environments, 
individual Unix vendors, such as SGI, Sun, HP, and IBM, added them to their own 
Unix variants. But without a true standards document, they fell into their old evil 
ways and added them with different specifications. Then along came Linux.... 

Linux changed everything. In many ways, the old joke is true: Linux is the Unix 
defragmentation tool. 3 As Linux became more popular, programs originally written 
for other Unixes were first ported to it, and then after a while were written for it and 
then ported to other platforms. This happened to Samba. Sun s SunOS on a SPARC 
system was, at first, our primary user platform, but after five years or so we rapidly 
migrated to Linux on Intel x86 systems. We now develop almost exclusively on 
Linux, and from there port to other Unix systems. 

This means the Linux interfaces are starting to take over as the most important stan 
dards for Unix-like systems to follow, in some ways supplanting POSIX and the SUS. 
The ACL implementation for Linux was added into the system, at first via a patch by 
Andreas Griinbacher, held externally to the main kernel tree. Finally it was adopted 
by the main Linux vendors, SuSE (now Novell) and Red Hat, and has become part of 
the official kernel. Other free Unix systems such as FreeBSD quickly followed with 
their own implementation of the last draft of the POSIX ACL specification, and now 
there are desktop GUI and other application programs that use the Linux ACL inter 
faces. As this code is ported to other systems, the pressure is on them to conform to 
the Linux APIs, not to any standards document. Sun has announced that its Solaris 
10 on Intel release will run Linux applications "better than Linux" and will be fully 
compatible at the system call level with Linux applications. This means Sun must 
have mapped the Linux ACL interface onto the Solaris one. Is that a good thing? 

In a world where Linux is rapidly becoming the dominant version of Unix, does 
POSIX still have relevance, or should we just assume Linux is the new POSIX? 



3 This was inspired by novice system administrators coming to Unix from the Windows platform 
for the first time and asking "where is the system defragmentation tool?", the concept of a 
filesystem designed well enough not to need one being outside their experience. 

Wither POSIX? JJ 45 



The Win32 (Windows) Standard 

Win32 was named for an expansion of the older Microsoft Windows interface, 
renamed the Win 16 interface once Microsoft was shipping credible 32-bit systems. 1 
have a confession to make. In my career, I completely ignored the original 16-bit 
Windows on MS-DOS. At that time, I was already working on sane 32-bit systems 
(68000 based), and dealing with the original insane 8086 segmented architecture 
was too painful to contemplate. Win32 was Microsoft s attempt to move the older 
architecture beyond the limitations of MS-DOS and into something that could com 
pete with Unix systems and to a large extent Microsoft succeeded spectacularly. 

The original 16-bit Windows API added a common GUI on top of MS-DOS, and also 
abstracted out the lower-level MS-DOS interfaces so that application code had a 
much cleaner "C" interface to operating system services (not that MS-DOS provided 
many of those). The Win32 Windows API was actually the "application" level API 
(not the system call level; I ll discuss that in a moment) for a completely new operat 
ing system that would soon be known as Windows NT ("New Technology"). This 
new system was designed and implemented by Dave Cutler, the architect of Digital 
Equipment Corporation s VMS system, long a competitor to Unix. It does share some 
similarities with VMS. The interface choice for applications was very interesting, sit 
ting on top of a system call interface that looks like Figure 3-2. 




Figure 3-2. Architecture of the Win32 API 

The idea behind the Windows NT kernel was that it could host several "sub 
system" system call interfaces, providing completely different application behavior 
from the same underlying kernel. It was meant to be a completely customizable 
operating system, providing different kernel "personalities" any ISV might require. 
The DOS subsystem and the (not-shown) 16-bit Windows subsystem were essen 
tial, as they provided backward compatibility for applications running on MS-DOS 
and 16-bit Windows; the new operating system would have gathered little accep 
tance had it not been able to run all the old MS-DOS and Windows applications. 



46 



A Tale of Two Standards 



The OS/2 subsystem was designed to allow users of text mode OS/2 applications 
(which was at one time a Microsoft product) to port them to Windows NT. 

The two interesting subsystems are the original POSIX subsystem and the new Win32 
subsystem. The POSIX subsystem was added, as the POSIX standard had become very 
prevalent in procurement contracts. Many of these valuable contracts were available 
only to systems that passed the POSIX conformance tests. So Microsoft added a mini 
mal POSIX subsystem into the new Windows NT operating system. This original sub 
system was, I think it s fair to say, deliberately crippled to make it unuseful for real- 
world applications: applications using it had no network access and no GUI access, so 
although a POSIX-compliant system might be required in a procurement contract, 
there usually was no requirement that the applications running on that system also had 
to be POSIX compliant. This allowed new applications using the Microsoft-preferred 
Win32 subsystem to be used instead. All might not have been lost if Microsoft had 
documented the internal subsystem interface, allowing third-party ISVs to create their 
own Windows NT kernel subsystems, but Microsoft kept this valuable information to 
itself (there was an exception to this, which I ll discuss shortly). 

So, let s examine the Win32 standard API, the interface designed to run on top of the 
Win32 kernel subsystem. It would be logical to assume that, like the POSIX system 
calls, the calls defined in the Win32 API would closely map to kernel-level Win32 
subsystem system calls. But that would be incorrect. It turns out that, when released, 
the Win32 subsystem system call interface was completely undocumented. The calls 
made from the application-level Win32 API were translated, via various shared 
libraries (DLLs in Windows parlance) mainly the NTDLLDLL library into the real 
Win32 subsystem system calls. 

Why do this, one might ask? Well, the official reasoning is that it allows Microsoft to 
tune and modify the system call layer at will, improving performance and adding fea 
tures without being forced to provide backward compatibility application binary inter 
faces (or ABIs for short). The more nefarious reasoning is that it allows Microsoft appli 
cations to cheat, and call directly into the undocumented Win32 subsystem system call 
interface to provide services that competing applications cannot. Several Microsoft 
applications were subsequently discovered to be doing just that, of course. One must 
always remember that Microsoft is not just an operating system vendor, but also the 
primary vendor of applications that run on its platforms. These days, this is less of a 
problem, as there are several books that document this system call layer, and there are 
several applications that allow snooping on any Windows NT kernel calls made by 
applications, allowing any changes in this layer to be quickly discovered and pub 
lished. But it left a nasty taste in the mouths of many early Windows NT developers 
(myself included). 



The Win32 (Windows) Standard X 47 



The original Win32 application interface was, on the surface, very well documented 
and cheaply available in paper form (five books at only $20 each; a bargain com 
pared to a POSIX specification). Like most things in Windows, on the surface it looks 
great. It covers much more than POSIX tries to standardize, and so offers flexible 
interfaces for manipulating the GUI, graphics, sound, and pen computing, as well as 
all the standard system services such as file I/O, file locking, threading, and security. 
Then you start to program with it. If you re used to the POSIX specifications, you 
almost immediately notice something is different. The details are missing. It s fuzzy 
on the details. You notice it the first time you call an API at runtime, and it returns 
an error that s not listed anywhere in the API documentation. "That s funny...." you 
think. With POSIX, all possible errors are listed in the return codes section of the API 
call. In Win32, the errors are a "rough guide." 

The lack of detail is one of the reasons that the Wine project finds it difficult to cre 
ate a working implementation of the Win32 API on Linux. How do you know when 
it s done? Remember that Linus, with some help, was able to create a decent POSIX 
implementation within a few years. The poor Wine developers have been laboring at 
this for 12 years, and it s still not finished. There s always one more wrinkle, one 
more undocumented behavior that some critical application depends on. Reminds 
me of Samba somehow, and for very similar reasons. 

It s not entirely Microsoft s fault. It hasn t documented its API because it hasn t 
needed to. POSIX was documented in detail due to need: the need of the developers 
creating implementations of the standard. Microsoft knows that whatever it makes 
the API do in the next service pack, that s still the Win32 standard. "Wherever you 
go, there you are," so to speak. 

However, the Win32 design does some things very well; security, for instance. Secu 
rity isn t the number one thing people think of when considering Windows, but in 
the Win32 API, security is a very great concern. In Win32, every object can be 
secured, and a property called a Security Descriptor, which contains an ACL, can be 
attached to it. This means objects such as processes, files, directories, and even 
Windows can have ACLs attached. This is much cleaner than POSIX, in which only 
objects in the filesystem can have ACLs attached to them. 

So, let s look at a Win32 ACL. As in POSIX, all users and groups are identified by a 
unique identifier. On POSIX, it s a uid_t type for users, and a gid_t type for groups. 
In Win32, both are of type SID or security identifier. A process or thread in Win32 
has a token attached to it that lists the primary SID of the process owner and a list of 
secondary group SID entries this user belongs to. Like in POSIX, this is attached to a 
process at creation time and the owner can t modify it to give himself more privi 
leges. A Win32 ACL consists of a list of SID entries with an attached bit mask identi 
fying the operations this SID entry allows or denies. Sounds reasonable, right? But 
the devil is in the details (see Figure 3-3). 

48 X A Tale of Two Standards 



Win32 process token 
Owner: SID 

Group: SID1 
List: 



Win32 security desciptor 




Owner: SID 

Primary group: SID-A I 

Access control list: 

Deny:[f*]:<bitmask>: SID 8 
Allow: [P]:<bitmask>: SID C 
Allow: ff]:<bitmask>: SID D 



[f*] represents flags for this entry 



Figure 3-3. Win32 access control 



Each SID entry in an ACL can be an allow entry or a deny entry. Their order is impor 
tant. Reorder a list of entries and swap a deny entry with an allow entry, and the mean 
ing of the ACL can change completely. POSIX ACLs don t have that problem because 
the evaluation algorithm defines the order in which entries are examined. In addition, 
the flags defining the entry (marked as [f*] in Figure 3-3) control whether an entry is 
inherited when the ACL is attached to a "container object" (such as a directory in the 
filesystem) and may also affect other attributes of this particular entry. 

The bit mask enumerates the permissions that this entry allows or denies. But the 
permissions are (naturally) different, depending on what object the ACL is attached 
to. Let s look at the kinds of permissions available for a file object: 

DELETE 

Delete the object. 

READ_CONTROL 

Read the ACL on an object. 

WRITE_DAC 

Write the ACL on an object. 

FILE_READ_DATA 

Read from the file. 

FILE_READ ATTRIBUTES 
Read file metadata. 

FILE_READ_EA 

Read extended attributes (if the file has any). 

FILE_WRITE_DATA 
Write to the file. 

FILE_WRITE_EA 

Write extended attributes (if the file has any). 



The Win32 (Windows) Standard X 49 



FILE_EXECUTE 

Open for execute (why do we need the . EXE tag then?). 

SYNCHRONIZE 

A permission related to an open file handle, not the file. 

And this is one of the simpler kinds of permission-bearing objects in Win32. 

If the Win32 API treats security so seriously, why does Windows fail most security 
tests in the real world? The answer is that most applications ignore this wonderful, 
flexible security mechanism because it s just too hard to use just like the problem 
with the POSIX pathconf ( ) call. No one can use the security mechanism correctly; 
applications would degenerate into a mess. It doesn t help that Microsoft, having 
realized the APIs controlling security were too difficult to use, keeps adding func 
tions to simplify this mess, sometimes also adding new APIs with a new service pack. 
In addition, as Microsoft has moved in the "Active Directory" world, it has extended 
the underlying semantics of the security mechanism, adding new flags and behaviors. 

Try taking a look at the "file security dialog" in Windows 2000. It s incomprehensi 
ble. No one, especially a system administrator, can keep track of this level of detail 
across their files. Everyone just sets one default ACL on the root of a directory hierar 
chy and hopes for the best. Most administrators usually want to do two simple things 
with an ACL: allow group X but not user Y, and allow group X and also user Z. This 
is just about comprehensible with POSIX ACLs, although those are near the limit of 
complexity that people can deal with. The Win32 security system is orders of magni 
tude more complex than that; it s hopelessly overdesigned. Computer scientists love 
it, as it s possible to do elegant little proofs of how secure it is, but in the real world, 
it s simply too much to deal with effectively great idea, adding ACLs to every sys 
tem object, but a real shame about the execution. 

Just to spread the blame around, the networking "experts" who designed the latest ver 
sion of Sun s network filesystem, NFS version 4, fell in love with this security mecha 
nism and decided it would be a great idea to add it into the NFSv4 specification. They 
probably thought it would make interoperability with Windows easier. Of course, they 
didn t notice that Microsoft had been busily extending the security mechanism as Win 
dows has developed, so they standardized on an old version of the Windows ACL 
mechanism, as Microsoft documented it (not as it actually works). So now, the Unix 
world has to deal with this mess or rather, a new network filesystem with an ACL 
model that is almost, but not quite, compatible with Windows ACLs, and that is com 
pletely alien to anything currently found on Unix. I sometimes feel Unix programmers 
are their own worst enemies. 



SO X A Tale of Two Standards 



The Tar Pit: Backward Compatibility 

Now, as an example of where Win32 got things spectacularly wrong, 1 want to look at 
a horror from the past that unfortunately got added into the Win32 interfaces due to 
the MS-DOS heritage. My pet hate with Win32 is the idea of "share modes" on open 
files. In my opinion, this one single legacy design decision has probably done more 
than any other to hold back the development of cluster-aware network filesystems on 
Win32 systems. 

Under POSIX, an open( ) call is very simple. It takes a pathname to open, the way in 
which you want to access or create the file (read, write, or both with various create 
types), and a permission mask that gets applied to files you do create. Under Win32, 
the equivalent call, CreateFile( ), takes seven parameters, and the interactions among 
them can be ferociously complex. The parameter that causes all the trouble is the 
ShareMode parameter, which can take values of any of the following constants OR ed 
together: 

FILE_SHARE_READ 

Allow others to open for read. 

FILE_SHARE_WRITE 

Allow others to open for write. 

FILE_SHARE_NONE 

Don t allow any other opens. 

FILE_SHARE_DELETE 

Allow open for delete intent. 

To make these semantics work, any Windows kernel dealing with an open file has to 
know about every other application on the system that might have this file open. 
This was fine back in the single-machine MS-DOS days, when these semantics were 
first designed, but it is a complete disaster when dealing with a clustered filesystem in 
which a multitude of connected file servers may want to give remote access to the 
same file, even if they serve out the file read-only to applications. They have to con 
sult some kind of distributed lock management system to keep these MS-DOS-inher 
ited semantics working. While this can be done, it complicates the job enormously 
and means cluster communication on every CreateFile( ) and CloseHandle( ) Zcall. 

This is the bane of backward compatibility. This idea of "share modes" arbitrating what 
access concurrent applications can have to a file is the cause of many troubles on a 
Windows system. Ever wonder why Windows has a mechanism built in to allow an 
application to schedule a file to be moved, but only after a reboot? Share modes in 
action. Why are some files on a Windows server system impossible to back up due to 
"another program is currently using this file" errors? Share modes again. There is no 
security permission that can prevent a user from opening a file with, effectively, "deny 



The Tar Pit: Backward Compatibility * 51 



all" permissions. If you can open the file for read access, you can get a share mode on 
it, by design. Consider a network-shared copy of Microsoft Office. Any user must be 
able to open the file WINWORD.EXE (the binary file containing Microsoft Word) to exe 
cute it. Given these semantics, any user can open the file with READ_DATA access with 
the ShareMode parameter set to FILE_SHARE_NONE and thus block use of the file, even 
over the network. Imagine on a Unix system, being able to open the /etc/passwd file 
with a share mode and deny all other processes access. Watch the system slowly grind 
to a halt as the other processes get stuck in this tar pit.... 

World Domination, Fast 

I ve heaped enough opprobrium on Win32. Let s give it a break and consider some 
thing the designers really did get right, and one of the advantages it has over POSIX. 
I m talking about the early adoption of the Unicode standard in Win32. When 
Microsoft was creating Win32, one of the things it realized was that this couldn t just 
be another English-only, American- and European-centric standard. It had to be able to 
not only cope with, but also encourage, applications written in all world languages 
(never accuse Microsoft of thinking small in its domination of the computing world). 

Given those criteria, its adoption of Unicode as the native character set for all the sys 
tem calls in Win32 was a stroke of genius. Even though the Asian countries aren t 
particularly fond of Unicode, because it merges several character sets they consider 
separate into one set of code points, Unicode is the best way to cope with the 
requirements of internationalization and localization in application development. 

To allow older MS-DOS and Win 16 applications to run, the Win32 API is available 
in two different forms, selectable by a compiler #def ine of -DUNICODE (it also helps if 
you own the compiler market for Windows, as Microsoft does, as you can standard 
ize tricks like this). The older code-page-based applications call Win32 libraries that 
internally convert any string arguments to 16-bit Unicode and then call the real 
Win32 library interface, which, like the Windows NT kernel, is Unicode only. 

In addition, Win32 comes with a full set of library interfaces to split out the text 
messages an application may need to display into resource files so that ISVs can eas 
ily have them translated for a target market. This eases the internationalization and 
localization burdens considerably for vendors. 

What is more useful, but not as obvious, is that making the Win32 standard natively 
use Unicode meant developers were immediately confronted with the requirements 
of multilingual code development. Many applications written in English-speaking (or 
Western European eight-bit character set-compatible) countries are badly written, 
making the assumption that a character will always fit within one byte. The early ver 
sions of Samba definitely made that mistake and retrofitting multibyte character set 



52 *J A Tale of Two Standards 



handling into old code is a real bear to get right. 1 know, because 1 was the person 
who first had to work on this for Samba (later I got some much-needed help from 
Andrew), so I may be a little touchy on this subject. 

Whenever I did Win32 development, I immediately designed with non-English lan 
guages in mind, and wrote everything with the abstract type TCHAR (one of the few 
useful abstract types in Win32), which is selectable at compile time using the Uni 
code defined to be either wchar_t with Unicode turned on, or unsigned char with 
Unicode turned off. Getting yourself in the right multibyte character set mindset 
from the beginning eliminates a whole class of bugs that you get when having to con 
vert a quick "English-only" hacked-up program into something maintainable for dif 
ferent languages. POSIX has been catching up over the years with the iconv( ) func 
tionality to cope with character set conversions, and Sun designed gettext( ) 
interfaces for localization, but Win32 had it all right from the start. 

Wither Win32? 

As with POSIX, the Win32 standard has not remained static over time. Microsoft has 
continued to develop and extend it, and has the advantage that anything it publishes 
immediately becomes the "standard," as is the case with all single vendor-defined 
standards. 

However, Microsoft is attempting to deemphasize Win32 as it moves into its new .NET 
environment and the new world of "managed code." Managed code is code running 
under the control of an underlying virtual machine (called the Common Language 
Infrastructure, or CLI, in .NET) and can be made to prevent the direct memory access 
that is the normal mode of operation of an API designed for C coding, such as Win32 
or POSIX. Free Software is also making a push into this area, with the Mono project, 
which implements the Microsoft C# language and .NET-managed code environment 
on Linux and other POSIX systems. 

Even if Microsoft is as successful as it hopes to be in pushing ISV programmers to 
convert to .NET and managed code using its new C# language, the legacy of applica 
tions developed in C using the Win32 API will linger for decades to come. ISV pro 
grammers are an ornery lot, especially people who have mastered the Win32 API, 
due to its less-than-complete documentation. 

What seems to happen over the years is that experienced Win32 programmers gain a 
sort of folk knowledge about the Win32 APIs i.e., how they really work versus 
what the documentation says. I often hang out on Usenet Windows discussion 
groups, and the attitudes of the experienced Windows programmers are very inter 
esting: they usually hate telling novices how stuff works. It s almost as if having 
learning Windows is a badge of honor, and they don t want to make earning that too 
easy for the neophytes. They exude an air of "they must suffer as I did." 



Wither Win32? X 53 



As Microsoft becomes less interested in Win32 with the release of its new Longhorn Win 
dows client and the move to managed code, is it possible for Microsoft to lose control of 
it? The POSIX standard is so complete because it was designed to allow programmers 
reading the standards documents to re-create a POSIX system from scratch. The Win32 
standard is nowhere near as well documented as that. However, there is hope in the 
Wine project, which is attempting to re-create a version of the Win32 API that is binary 
compatible with Windows on Intel x86 systems. Wine is, in effect, a second implementa 
tion of the Win32 system, making it closer to a true vendor-independent standard. 
Efforts taking place at companies such as Code Weavers and Transgaming Technologies 
are very promising; I just finished playing the new Windows-only game Half -Life 2 on my 
desktop Linux system, using the Wine technology. This is a significant achievement for 
the Wine code and bodes well for the future. 

Choosing a Standard 

BETWEEN TWO EVILS, I ALWAYS LIKE TO TAKE THE ONE I VE NEVER TRIED BEFORE. 

Mae West 

So, what should we choose when examining what standards to support and develop 
applications for? What should we recommend to businesses and governments that 
are starting to look closely at the open source/free software options available? 

It s important that businesses and governments selecting standards-based products 
pay attention to open standards. No more of the Microsoft Word .DOC format stan 
dard (which suffers from the same problem as Win32 in terms of it being single-ven 
dor controlled). No de facto vendor standards, no matter how convenient. They need 
to select standards that are at the same level as POSIX namely, standards to the 
level that other implementations can be created from the documentation. It s simple 
to tell when a standard meets that criterion because other implementations of it exist. 

The interesting thing is that both POSIX and Win32 standards are now available on 
both systems. On Linux, we have the POSIX standard as native, and the Wine project 
provides a binary-compatible layer for compiled Win32 programs that can run many 
popular Win32 applications. Perhaps more interestingly for programmers, the Wine 
project also includes a Linux shared library, winelib, which allows Win32 applica 
tions to be built from source code form on POSIX systems. What you end up with is 
an application that looks like a native Windows application, but can be run on non- 
Intel platforms; something that early versions of Windows NT used to support, but 
now is restricted to x86-compatible processors. Taking your Win32 application and 
porting it using winelib is an easy way to get your feet wet in the POSIX world, 
although it won t look like a native Linux application (this may be a positive thing if 
your users are used to a Windows look and feel). 



54 



If you ve already gone the .NET and C# route, using the Mono project may enable 
your code to run on POSIX systems. 

On Windows, there is now a full POSIX subsystem, supported by Microsoft and avail 
able for free. Earlier 1 alluded to Microsoft s reluctance to release information on how to 
create new subsystems for the Windows NT kernel, but it turns out that earlier in its 
history Microsoft was not so careful. A small San Francisco-based company, Softway 
Systems, licensed the documentation and produced a product called OpenNT (later 
renamed Interix), which was a replacement for Microsoft s originally crippled POSIX 
subsystem. Unfortunately, OpenNT didn t sell very well; someone cruelly referred to it 
as having "all the application availability of Linux, with the stability of Windows." As 
the company was failing, Microsoft bought it (probably to bring the real gem of the 
Windows kernel subsystem interface knowledge back in-house) and used it to create 
its Services for Unix (SFU) product. SFU contains a full POSIX environment, with a 
software development kit allowing applications to be written that have access to net 
working and GUI APIs. The applications written under it run as full peers with the 
mature Win32 applications, and users can t tell the difference. 

Recently Mcrosoft made SFU available as a free download to all Windows users. I 
like to think the free availability of Samba had something to do with this, but maybe 
I m flattering the Samba team too much. As I like to say in my talks, "If you re into 
piloting Samba on Linux in your organization, you re paying too much for your 
Microsoft software." But what this means is that if you want to write a completely 
portable application, the one standard you can count on to be there and fully imple 
mented and supported on Windows, Linux, Solaris, Apple Mac OS X, HP-UX, AIX, 
IRIX, and all the other Unix systems out there is POSIX. 

So, if you ll excuse me, I m going to look at porting parts of Samba to Windows.... 



Choosing a Standard ^C 55 





t CHAPTER 4 

Ben Laurie 




Open Source and Security 



More than two years ago, in a fit of frustration over the state of open source security, 
I wrote my first and only blog entry 1 (for O Reilly s Developer Weblogs): 

June and July were bad months for free software. First Apache chunked encod 
ing vulnerability, 2 and just when we d finished patching that, we get the 
OpenSSH hole. 3 Both of these are pretty scary the first making every single 
web server potentially exploitable, and the second makes every remotely man 
aged machine vulnerable. 

But we survived that, only to be hit just days later with the BIND resolver prob 
lems. 4 Would it ever end? Well, there was a brief respite, but then, at the end of 
July, we had the OpenSSL buffer overflows. 5 

All of these were pretty agonising, but it seems we got through it mostly unscathed, 
by releasing patches widely as soon as possible. Of course, this is painful for users 
and vendors alike, having to scramble to patch systems before exploits become 
available. I know that pain only too well: at The Bunker, 6 we had to use every avail- 



1 http://www.oreillynet.com/pub/wlg/2004. 

2 http://cve.mitre.org/cgi-bin/cvename. cgi?name=CVE-2002-0392. 

3 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2002-0639. 

4 http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2002-0651 . 

5 http://cve.mitre.org/cgi-biiticvename.cgi7name~CAN-2002-0656. 

6 Back in those days, The Bunker belonged to A.L. Digital Ltd., and it wasn t called The Bunker 
Secure Hosting. 





" 



able sysadmin for days on end to fix t he problems, which seemed to be arriving 
before we d had time to catch our breath from the previous one. 

But I also know the pain suffered by the discoverer of such problems, so I 
thought I d tell you a bit about that. First, I was involved in the Apache chunked 
encoding problem. That was pretty straightforward, because the vulnerability 
was released without any consultation with the Apache Software Foundation, a 
move I consider most ill advised, but it did at least simplify our options: we had 
to get a patch out as fast as possible. Even so, we thought we could take a little 
bit of time to produce a fix, since all we were looking at was a denial-of-service 
attack, and let s face it, Apache doesn t need bugs to suffer denial of service all 
this did was make it a little cheaper for the attacker to consume your resources. 

That is, until Gobbles 7 came out with the exploit for the problem. Now, this 
really is the worst possible position to be in. Not only is there an exploitable 
problem, but the first you know of it is when you see the exploit code. Then we 
really had to scramble. First we had to figure out how the exploit worked. I fig 
ured that out by attacking myself and running Apache under gdb. I have to say 
that the attack was rather marvelously cunning, and for a while I forgot the 
urgency of the problem while I unravelled its inner workings. Having worked 
that out, we were in a position to finally fix the problem, and also, perhaps more 
importantly, more generically prevent the problem from occurring again 
through a different route. Once we had done that, it was just a matter of writing 
the advisory, releasing the patches, and posting the advisory to the usual places. 

The OpenSSL problems were a rather different story. I found these whilst work 
ing on a security review of OpenSSL commissioned by DARPA 8 and the USAF. 9 
OpenSSL is a rather large and messy piece of code that I had, until DARPA 
funded it, hesitated to do a security review of, partly because it was a big job, 
but also partly because I was sure I was going to find stuff. And sure enough, I 
found problems (yes, I know this flies in the face of conventional wisdom 
many eyes may be a good thing, but most of those eyes are not trained observ 
ers, and the ones that are do not necessarily have the time or energy to check the 
code in the detail that is required). Not as many as I expected, but then, I 
haven t finished yet (and perhaps I never will, it does seem to be a never-ending 
process). Having found some problems, which were definitely exploitable, I was 
then faced with an agonising decision: release them and run the risk that I 
would find more, and force the world to go through the process of upgrading 
again, or sit on them until I d finished, and run the risk that someone else would 
discovered them and exploit them. 



7 A hacker (or group of hackers, it is not known which). 

8 The United States Defense Advanced Research Projects Agency, responsible for spending a great 
deal of money on national security in this case, for a thing known as CHATS, or Composable 
High Assurance Trusted Systems. 

9 Yes, I do mean the United States Air Force. 

SB X Open Source and Security 



In fact, I dithered on this question for at least a month then one of the prob 
lems I d found was fixed in the development version without even being noted 
as a security fix, and another was reported as a bug. I decided life was getting 
too dangerous and decided to release the advisory, complete or not. Now, you 
might think that not being under huge time pressure is a good thing, but in 
some ways it is not. The first problem came because various other members of 
the team thought I should involve various other security alerting mechanisms 
for example, CERT 10 or a mailing list operated by most of the free OS vendors. 11 
But there s a problem with this: CERT s process is slow and cumbersome and I 
was already nervous about delay. Vendor security lists are also dangerous 
because you can t really be sure who is reading them and what their real inter 
ests are. And, more deeply, I have to wonder why vendors should have the ben 
efit of early notification, when it is my view that they should arrange things so 
that their users could use my patches as easily as I can. I build almost every 
thing from original source, so patches tend to be very easy to use. RPMs 12 and 
ports 13 make this harder, and vendors who release no source at all clearly com 
pletely screw up their customers. Why should I help people who are getting in 
the way of the people who matter (i.e., the users of the software)? 

Then, to make matters worse, one of the more serious problems was reported 
independently to the OpenSSL team by CERT, who had been alerted by Defcon. 14 
I was going, and there was no way I was delaying release of the patches until after 
DeFcon. So, the day before I got on a plane, I finally released the advisory. And 
the rest is history. 

So, what s the point of all this? Well, the point is this: it was a complete waste of 
time. I needn t have agonised over CERT or delay or any of the rest of it. Because 
half the world didn t do a damn thing about the fact they were vulnerable, and 
because of that, as of yesterday, a worm is spreading through the Net like wild 
fire. 

Why do I bother? 

Two years later, I am still bothering, so I suppose that I do think there s some point. 
But there are interesting questions to ask about open source security is it really true 
that "many eyes" doesn t work? How do we evaluate claims about the respective vir 
tues of open and closed source security? Has anything changed in those two years? 
What is the future of open source security? 



10 CERT is an organization funded to characterize security issues and alert the appropriate 
parties a job they do not do very well, in my opinion. 

1 1 Apparently, I m not one, so I m not on this list. 

12 One of those recursive definitions programmers love: RPM Package Manager, a widely used 
system for distributing packaged open source software, particularly for various flavors of Linux. 

13 FreeBSD s package management system. Also used by other BSDs. 

14 DefCon is a popular hacker s convention, held annually in Las Vegas. 



* cq 

K1 33 



Many Eyes 

Although it s still often used as an argument, it seems quite clear to me that the 
"many eyes" argument, 15 when applied to security, is not true. It is worth remember 
ing what was originally said: "Many eyes make all bugs shallow" (Eric S. Raymond). I 
believe this is actually true, if read in the right context. Once you have found a bug, 
many eyes will, and indeed, do, make fixing it quick and easy. 

Security vulnerabilities are no different in this respect once they are found, they are 
generally easy to track down and fix (the Apache chunked encoding vulnerability was 
the hardest I ve ever had to track, and even that took only one long day s work). But 
vulnerabilities aren t like bugs in that sense until they are discovered. Once you find 
them, you have a recipe for making the software behave unexpectedly. Until that time, 
what do you have? A piece of software that does what you expect. 

The idea that bugs and security vulnerabilities are really the same thing is quite 
wrong and it s an idea that I suspect has been perpetrated by the reliability commu 
nity, 16 sensing a new source of funding. Software is reliable if it does what is 
expected when operated as expected. It is secure if it does what is expected under all 
circumstances. This is a very critical difference, indeed. Nonsecurity bugs have a sig 
nificant qualitative difference from security bugs people don t go out of their way to 
find bizarre things to do to make the software go wrong just for the fun of it. And if 
they do, and it s not a security hole... well, yes, that s interesting, and we ll fix it one 
day but, in the meantime, you didn t need that functionality, so just stop poking 
yourself in the eye and it will stop hurting. 

What has happened is that advocates of open source have taken the "many eyes" 
argument to mean that because the source is available, many people will examine it 
for weaknesses. This simply isn t true: most people never look at the source at all 
(until it doesn t work), and even if they do, most do not have the experience to find 
the problems. The argument simply does not hold water, and it s time we, as a com 
munity, abandon it. 

However, there is an important sense in which the "many eyes" theory holds a grain 
of truth: those who want to look at the source to check for vulnerabilities, can. The 
interesting question is whether those who want to look the the code are generally the 
good guys or the bad guys. But this is a question I will come to later, when I com 
pare open and closed source. 



15 The argument is that if enough people look at the code, bugs (and hence security issues) will be 
found before they bite you. 

16 Academics who study the reliability, as opposed to the security, of computer systems. 



60 *C Open Source and Security 



Open Versus Closed Source 

Since I wrote my rant, Microsoft has decided that security is important (at least for 
sales), and as a result, there s been a sudden increased interest in the truth of the 
claim that open source is "more secure" than closed source and, of course, the 
counterclaim of the opposite. 

But this claim is not easy to examine, for all sorts of reasons. First, what do we mean by 
"more secure"? We could mean that there are fewer security bugs, but surely we have 
to take severity of the bugs into account, and then we re being subjective. We could 
mean that when bugs are found, they get fixed faster, or they damage fewer people. Or 
we might not be talking about bugs at all. We might mean that the security properties 
of the system are better in some way, or that we can more easily evaluate our exposure 
to security problems. 

I expect that, at some point, almost everyone with a serious interest in this question will 
choose one of these definitions, and at some other point a completely different one. 

Who Is the Audience? 

It is also important to recognize that there are at least two completely different reasons to 
ask the question "is A more secure than B?" One is that you are trying to sell A to an 
audience that just wants to tick the "secure" box on their checklist, and the other is 
because you actually care about whether your product/web site/company/whatever is 
secure, and are in a position to have an informed opinion. 

It is, perhaps, unkind to split the audience in this way but, sadly, it appears to be a 
very real split. Most people, if asked whether they think the software they use should 
be secure will say, "Oh yeah, security, that s definitely a good thing, we want that." 
But this does not stop them from clicking Yes to the dialog box that says "Would you 
like me to install this Trojan now?" or running products with a widely known and 
truly dismal security record. 

However, it is a useful distinction to make. If you are trying to sell to an audience 
that wants to tick the security box, you will use quite different tactics than if the 
audience truly cares about security. This gives rise to the kind of analysis I see more 
and more. For example, http://dotnetjunkies.com/Weblog/stefandcmetz/archive/2004/10/ 
ll/28280.aspx has an article titled "Myth debunking: SQL Server vs. MySQL security 
2003-2004 (SQL Server has less bugs!!)." The first sentence of the article gives the 
game away: "Seems that yet again a MS product has less bugs that (sic) the corre 
sponding LAMP 17 product." What is this telling us? Someone found an example of a 
closed source product that is "better" at security than the corresponding open source 



17 LAMP stands for Linux, Apache, MySQL, Perl (or PHP) and is common shorthand for the 
cluster of open source commonly used to develop web sites. 



Open Versus Closed Source * * 61 



one. Therefore, all closed source products are "better" at security than open source 
products. If we keep on saying it, it must be true, right? 

Even if I ignore the obviously selective nature of this style of analysis, I still have to 
question the value of simply counting vulnerabilities. I know that if you do that, 
Apache appears to have a worse record than IIS recently (though not over longer 
periods). 

But I also know that the last few supposed vulnerabilities in Apache have been either 
simple denial-of-service (DoS) attacks 18 or vulnerabilities in obscure modules that 
very few people use. Certainly I didn t even bother to upgrade my servers for any of 
the last half-dozen or so; they simply weren t affected. 

So, for this kind of analysis to be meaningful, you have to get into classifying vulner 
abilities for severity. Unfortunately, there s not really any correct way to do this. 
Severity is in the eye of the beholder. For example, my standard threat model (i.e., 
the one I use for my own servers, and generally advise my clients to use, at least as a 
basis) is that all local users 19 have root, 20 whether you gave it to them or not. So, 
local vulnerabilities 21 are not vulnerabilities at all in my threat model. But, of course, 
not everyone sees it that way. Some think they can control local users, so to them, 
these holes matter. 

Incidentally, you might wonder why I dismiss DoS attacks; that is because it is essen 
tially impossible to prevent DoS attacks, even on perfectly functioning servers, since 
their function is to provide a service available to all, and simply using that service 
enough will cause a DoS. They are unavoidable, as people subject to sustained DoS 
attacks know to their pain. 

Time to Fix 

Another measure that I consider quite revealing is "time to fix" that is, the time 
between a vulnerability becoming known and a fix for it coming available. There are 
really two distinct measures here, because we must differentiate between private and 
public disclosure. If a problem is disclosed only to the "vendor," 22 the vendor has the 
leisure to take time fixing it, bearing in mind that if one person found it, so will others 



18 In a DoS attack, the attacker prevents access by legitimate users of a service by loading the 
service so heavily that it cannot handle the demand. This is often achieved by a distributed 
denial of service (DDoS) attack, in which the attacker uses a network of "owned" (i.e., under the 
control of the attacker and not the legitimate owner) machines to simultaneously attack the 
victim s server. 

19 That is, people with user accounts on the machine, rather than visitors to web pages or people 
with mail accounts, for example. 

20 Root is the all-powerful administrative account on a Unix machine. 

21 A local vulnerability is one that only a local user can exploit. 

22 A term I am not at all fond of, since, although I am described as a "vendor" of Apache, 
OpenSSL, and so forth, I ve never sold any of them. 



62 X Open Source and Security 



meaning "leisure" is not the same as "forever," as some vendors tend to think. The time 
to fix then becomes a matter of negotiation between vendor and discloser (an example of 
a reasonably widely accepted set of guidelines for disclosure can be found at http://www. 
wiretrip.net/rfp/policy.html, though the guidelines are not, by any means, universally 
accepted) and really isn t of huge significance in any case, because the fix and the bug 
will be revealed simultaneously. 

What is interesting to measure is the time between public disclosures (also known as 
zero-days) and the corresponding fixes. What we find here is quite interesting. Some 
groups care about security a lot more than others! Apache, for example, has never, to 
my knowledge, taken more than a day to fix such a problem, but Gaim 23 recently left 
a widely known security hole open for more than a month. Perhaps the most inter 
esting thing is that whenever time to fix is studied, we see commercial vendors Sun 
and Microsoft, for example pitted against open source packagers Red Hat and 
Debian and the like but this very much distorts the picture. Packagers will almost 
always be slower than the authors of the software, for the obvious reason that they 
can t make their packages until the authors have released the fix. 

This leads to another area of debate. A key difference between open and closed 
source is the number of "vendors" a package has. Generally, closed source has but a 
single vendor, but because of the current trend towards packagers of open source, 
any particular piece of software appears, to the public anyway, to have many differ 
ent vendors. This leads to an unfortunate situation: open source packagers would 
like to be able to release their packages at the same time as the authors of the pack 
ages. I ve never been happy with this idea, for a variety of reasons. First, there are so 
many packagers that it is very difficult to convince myself that they will keep the 
details of the problem secret, which is critical if the users are not to be exposed to the 
Bad Guys. Second, how do you define what a packager is? It appears that the critical 
test I am supposed to apply is whether they make money from packaging or not! 24 
This is not only blatantly unfair, but it also flies in the face of what open source is all 
about. Why should the person who participates fully in the open source process by 
building from source be penalized in favor of mere middlemen who encourage peo 
ple not to participate? 25 

Of course, the argument, then, is that I should care more about packagers because if 
they are vulnerable, it affects more people. I should choose whom I involve in the 
release process on the basis of how many actual users will be affected, either posi- 



23 A popular open source instant messaging client. 

24 Of course, not all packagers make money, but I ve only experienced this kind of pressure from 
those that do. 

25 This is because vendors tend to encourage users to treat them as traditional closed source 
businesses with their own support, their own versions of software, and so forth instead of 
engaging the users with the actual authors of the software they are using. 



Open Versus Closed Source *C 63 



lively or negatively, depending on whether I include the packager or not, by my 
choice. I should also take into account the importance of these users. A recent argu 
ment has been that I should involve organizations such as the National Infrastruc 
ture Security Co-ordination Centre (NISCC), a UK body that does pretty much what 
it says on the tin, and runs the UK CERT (see http://www.niscc.gov.ufe for more infor 
mation) because they represent users of more critical importance than mere mortals. 
This is an argument I actually have some sympathy with. After all, I also depend on 
our infrastructure. But in practice, we soon become mired in vested interests and 
commercial considerations because, guess what? Our infrastructure uses software 
from packagers of various kinds, so obviously I must protect the bottom line by mak 
ing sure they don t look to be lagging behind these strange people who give away 
security fixes to just anyone. 

If these people really cared about users, they would be working to find ways that 
enable the users to get the fixes directly from the authors, without needing the pack 
ager to get its act together before the user can have a fix. But they don t, of course. 
They care about their bank balance, which is the saddest thing about security today: 
it is seen as a source of revenue, not an obligation. 

Incidentally, a recent Forrester Research report claims that packagers are actually 
quite slow as slow as or slower than closed source companies at getting out fixes. 
This doesn t surprise me, because a packager generally has to wait for the (fast!) 
response of the authors before doing its own thing. 

Visibility of Bugs and Changes 

There is argument that lack of source is actually a virtue for security. Potential attack 
ers can t examine it for bugs, and when vulnerabilities are found, they can t see what, 
exactly, was changed. 

The idea that vulnerabilities are found by looking at the source is an attractive one, 
but is not really borne out by what we see in the real world. For a start, reading the 
source to anything substantial is really hard work. I know I did it for OpenSSL, as I 
said earlier. In fact, vulnerabilities are usually found when software misbehaves, 
given unusual input or environment. The attacker follows up, investigating why that 
misbehavior occurred and using the bug thus revealed for their own evil ends. The 
"chunked encoding" bug I mentioned earlier is a great example of this. This was 
found by the common practice of feeding programs large numbers of the same char 
acter repeatedly. When Apache was fed A lots of times, it ended up treating it as a 
count of characters in hex, and it came out negative, which turns out to be a Bad 
Thing. In this case, all that was needed was eight characters, but the problem was 
found by feeding Apache several thousand. 26 



26 This particular method is popular because it is so easy: perl -e "print "A xlOOOO" | target. 
64 xx Open Source and Security 



So, not having the source might slow down an attacker slightly, but given the avail 
ability of excellent tools like IDA (a very capable disassembler) and Ollydbg (a pow 
erful [and free! debugger), not by very much. 

What about updates? The argument is that when source is available, the attacker can 
compare the old and new versions of the source to see what has changed, and then use 
that to craft software that can exploit unfixed versions of the package. In fact, because 
most open source uses version control software, and often has an ethos of checking in 
changes that are as small as possible, usually the attacker can find just the exact 
changes that fixed the problem without any clutter arising from unrelated changes. 

But does this argument hold water? Not really, as, for example, Halvar Flake has 
demonstrated very clearly with his Binary Difference Analysis tool. What this does is 
take two versions of a program, before and after a fix, disassembles them, and then 
uses graph isomorphisms to work out what has changed. I ve seen this tool in action, 
and it is very impressive. Halvar claims (and I believe him) that he can have an 
exploit out for a patched binary in one to eight hours from seeing the new version. 

Review 

Another important aspect to security is the ability to assess the risks. With closed 
source, this can be done only on the basis of history and reputation, but with open 
source, it is possible to go and look for yourself. Although you are not likely to find 
bugs this way, as I stated earlier, you can get a good idea about the quality of the 
code, the way it has been written, and how careful the author is about security. And, 
of course, you still have history and reputation to aid you. 

Who s the Boss? 

Finally, probably the most important thing about open source is the issue of who is 
in control. When a security problem is found, what happens if the author doesn t fix 
it? If the product is a closed source one, that generally is that. The user is doomed. 
He must either stop using it, find a way around the problem, or remain vulnerable. 
In contrast, with open source, users are never at the mercy of the maintainer. They 
can always fix the problem themselves. 

It is often argued that this isn t a real choice for end users; usually end users are not 
programmers, so they cannot fix these problems themselves. This is true, but it com 
pletely misses the point. Just as the average driver isn t a car mechanic but still has a 
reasonably free choice of who fixes his car, 27 he can also choose a software maintainer 
to fix his software for him. In practice, this is rarely needed because (at least for any 
widely used software) there s almost always someone willing to take on the task. 



27 This is a metaphor that is rapidly going out-of-date, as car manufacturers make cars more and 
more computerized and harder and harder for anyone not sanctioned by the manufacturer to 
work on. Who knows perhaps this will lead to an open source culture in the car world. 



Open Versus Closed Source X 65 



Digression: Threat Models 

I mentioned threat models earlier. Because not all my readers will be security experts, 
it is worth spending a moment to explain what I mean. When you evaluate a threat 
to your systems, you have to have a context in which to do it. Simply saying "I have a 
security hole" tells you almost nothing useful about it. What you want to know is 
how bad it is, how fast you have to fix it, what it will cost if you don t fix it, and what 
it will cost if you do fix it. 

To make that assessment, there are various things you need to know. The obvious 
ones are what systems you are running; what the value of each component is; what 
impact the vulnerability will have on each component; how likely you are to be 
attacked; and so forth. But less obvious is the question of whether you actually care 
about the attack at all and this is where threat models come in. They characterize 
what you have already assumed yourself to be vulnerable to and how you are vulner 
able it. 

So, as I mentioned, my threat model is that local users have root. Because root can 
do, essentially, anything she wants, this means that any vulnerability that can only be 
exploited by a local user, no matter what it is, and no matter how bad, is irrelevant to 
me. They could do that already. 

Threat models can get quite complicated, and you may well find that when a new 
vulnerability comes along, you have to consider what your model actually is, because 
you don t already know. For example, suppose there s an attack on the domain name 
service that allows it to be faked. Do you care? Was that something you assumed had 
to be correct when you built your system, or is incorrectness merely a nuisance? 

Anyway, I don t want to turn this chapter into a textbook on security, so suffice it to 
say that threat models are important, everyone s is different, and you can t evaluate 
the impact of vulnerabilities without one which means, really, that the whole ques 
tion of which is better is one only you can answer. 

The Future 

PREDICTION is DIFFICULT, ESPECIALLY ABOUT THE FUTURE. 

Niels Bohr/Mark Twain 28 

There are two futures: the one we should have, and the one we re going to get. I ll 
talk about the one we should have first, because it s more fun, more interesting, and 
definitely more secure. 



28 Apparently it s difficult about the past too we don t know which of these people said this! 
66 X Open Source and Security 



Today s operating systems and software are based on decades of experience with 
developing software that was run by nice guys on machines over which they con 
trolled access relatively easily (whether as users or nonusers interacting with the 
machine or software in some way). This was a world where your biggest security 
threat was a student playing a prank. We learned a great deal about how to write 
software that did clever things, was easy to use, and had pretty interfaces. 

Unfortunately, we learned almost nothing about how to write secure software. And 
in the meantime, we built up a huge amount of insecure software. Worse, we used 
insecure languages to write the insecure software in. And worse even than that, we 
used languages thath there s no real prospect of securing. And we continue to use 
them, and the same insecure operating systems we wrote, with ever-increasing teeter 
ing towers of software piled on top of them. 

So, in my Brave New World, we get smart enough to scrap all this and use an idea 
invented in the 1960s: capabilities. Unfortunately, academics decided very early on that 
capabilities had all sorts of problems, and this has prevented their widespread adoption. 
Mark Miller and Jon Shapiro, in "Paradigm Regained: Abstraction Mechanisms for 
Access Control" (http://w\vw.erights.org/talks/asian03/paradigm-revised.pdf), have very 
effectively debunked these criticisms, though I have to admit to being bemused by how 
anyone could believe them in the first place, since they are so easily solved. 

In any case, there are still some of us around who believe in capabilities, and I enter 
tain the fond hope that we may start using them on a larger scale. The foremost 
project using capabilities at the moment is the E language (http://www.erights.org), 
which, as well as being a capability language from the ground up, has some very nice 
features for distributed computing, and is well worth a look. Unfortunately, I do not 
believe a language with such esoteric (and ever-changing) syntax will ever be widely 
used. It seems that privilege belongs to a very few. Perhaps more promising from the 
point of view of likelihood of adoption is my own nascent CAPerl (think "Kapow!") 
project, which adds capabilities to Perl. Although this is far less elegant and satisfy 
ing, it has the virtue of looking almost exactly like Perl to the experienced program 
mer, and so I do have some hope that it might actually get used. I don t have a web 
site for it yet, so I invite you to Google for it. 

No discussion of capabilities in the 21st century would be complete without men 
tioning EROS (http://www.eros-os.org). Funnily enough, EROS is short for Extremely 
Reliable Operating System, since its author, Jon Shapiro, thought that was what was 
important about it when he started writing it. Now, though, we are far more inter 
ested in its security properties than in its reliability. EROS, like E, implements capa 
bilities from the ground up. More importantly, it runs on PCs. Unfortunately, it 
seems it is a project that won t be finished. Work is, however, starting soon on the 
second attempt. 



The Future JJ 67 



Of course, if I really think this will happen, I m on crack. Not enough people care 
enough about security to contemplate throwing everything away and starting again 
(make no mistake, that s what it takes). But I can (and do) hope that people will start 
writing new things using capabilities. And I hope that drawing them to your atten 
tion will assist that. 

Now I ll move on to what I think will really happen. Certainly people have become 
more aware of security as an issue, and the increasing use of open source in corpo 
rate environments also increases the pressure on security. It seems likely that this will 
drive open source toward better ways to deliver updates faster. I don t think it is 
actually possible to drastically improve open source s record on fixing security issues. 
I believe that by any measure, that open source is ahead of closed source. But the 
flow from author to end user is not yet a smooth one. 

Interestingly, the fix for that is strongly related to the fix for another widely acknowl 
edged problem with open source: package management systems. We do not yet have 
the ultra-smooth systems to handle installation and update of systems in a way that 
makes it a no-brainer for end users. Open source and closed source present interest 
ingly different problems. Open source packages of any complexity tend to depend on 
other open source packages, usually with a completely different set of authors and 
release cycles. Managing installation in this environment is much harder than in the 
closed source situation, where one vendor even one that buys components from 
others is in control of the whole package. I think the open source world is moving 
toward better package management, and this will automatically improve the end 
user s management of security. 

However, for corporate environments, this probably makes little difference. In such 
situations, there are almost always elaborate procedures for rolling out new versions 
which are almost unchanged when using open source. Even so, clearer visibility of 
dependencies and, therefore, what needs to be upgraded when a fix comes out, 
would be useful. 

I also hope that better package management would reduce the dependency of users 
(at least, if they choose to have their dependency reduced) on packagers. Although 
packagers, in theory, add value, they also add latency. Perhaps worse, they damage 
the open source model by introducing dozens of slightly different versions of each 
package, through the widespread practice of applying patches to the packages 
instead of contributing them back to the original authors, which reduces the effec 
tiveness of community development by splitting the community into many smaller 
subcommunities . 

As always, there is a price to be paid for better package management. Automated 
updates are a fantastic vector to mount automated attacks. We know well how to 
prevent such attacks using public key cryptography, but once more, the complexity 



88 X Open Source and Security 



of multiple authors introduces problems of key management to which there aren t 
really good answers, at least, so far. 29 

One thing that does seem certain is that the increasing trend of concern about secu 
rity by end users will continue. The seemingly never-ending rise of spam, adware, 
and Trojans, if nothing else, has put it on everyone s agenda, and that doesn t seem 
likely to change. 

Interesting Projects 

I ve already mentioned some projects in passing, but no chapter on open source 
security would be complete without mentioning some of the more interesting 
projects out there. I ll start with the obvious ones and move on to the more esoteric. 
This list probably reflects my current obsession with privacy and anonymity: 

OpenSSL 

Well known, but still essential. This library implements most known crypto 
graphic algorithms, as well as the SSL and TLS protocols. It is very widely used 
in both free and non-free software, and at the time of this writing was in the final 
stages of obtaining FIPS-140 certification, http://www.openssl.org. 

Apache 2 

Of course, we ve all known and loved Apache for years. Finally, Apache 2 has 
HTTPS support out of the box. http://www.apache.org. 

Mozilla 

A suite of web browser, mail, and news reading software, and related utilities. 
You probably don t think of this as security software, but it is probably second 
only to Apache in the number of financial transactions it protects. And it does it 
with a minimum of fuss. What s more, it isn t plagued with its closed-source 
rivals fondness for installing evil software you never intended to install! http.7/ 
www.mozilla.org. 

GnuPG 

Implementing the OpenPGP standard under the GPL. Primarily used for email, 
but also the mainstay for validation of open source packages (using, of course, 
public key cryptography), http://www.gnupg.org. 

Emgmail 

Small, but (almost) perfectly formed. This is a plug-in for the increasingly popu 
lar (and, of course, open source) email client, Thunderbird, providing a nicely 
streamlined interface for GnuPG. http://enigmail.mo2dev.org. 



29 1 should perhapsat this point plug KeyMan, a package I designed to solve this problem, but 
since it has singularly failed to take off, that might be inappropriate. 



Interesting Projects X 69 



CVE 

Common Vulnerabilities and Exposures. This is a database of security problems, 
both commercial and open source. The idea is to provide a uniform reference for 
each problem, so it s easy to tell if two different people are talking about the 
same bug. http://cve.mitre.org. 

TOR 

The onion router. Onion routing has been a theoretical possibility for a long 
time, providing a way to make arbitrary connections anonymously. Zero Knowl 
edge Systems spectacularly failed to exploit it commercially, but now it has come 
from a most unlikely source: the U.S. Navy. The Navy s funding recently ran out, 
but the Electronic Frontier Foundation stepped up to take over. Well worth a 
look, http://tor.eff.org. 

Conclusion 

In the end, it seems to me there s little to be sensibly said that, from the viewpoint of 
security, truly differentiates between open and closed source. The points I believe are 
critical are my ability to review the code for myself and my ability to fix it myself when 
it is broken. By "myself I do, of course, include "or anyone of my choice." What I don t 
believe in at all is the often-quoted but never-proven "many eyes" theory. 

In the digression on threat models, I mentioned that the only person who can really 
answer the question of whether open source is better for security is you. Leave the 
camp of people who think security is a good thing that we should all have more of, 
and join the camp of people who have thought about what it means to them, what 
they value, and so, what they choose. 



70 X Open Source and Security 





b CHAPTER 5 

Michael Olson 




Dual Licensing 



Over the past decade, there have been many attempts to commercialize open source 
software. One common strategy has been to create services businesses, which offer 
consulting and support to users of open source. Another strategy has been to build 
hybrid businesses, which distribute open source platforms with proprietary add-ons, 
and which make money by licensing the add-ons. 

A third strategy, and the focus of this chapter, is called dual licensing. Companies that 
use dual licensing provide a single software product under two different licenses. 
One license, which imposes open source terms, is available to a certain class of users. 
A second license, with proprietary terms, is available to others. 

Business and Politics 

This chapter is about business. Software is deep in the modern economy: it provides 
the mechanism for the flow of capital around the world, and it is itself a good that 
can be produced, bought, and sold. Whenever something interesting happens in the 
world of software, business leaders pay attention. 

Open source is interesting. It enforces new rules for use and distribution of software 
products. It changes the economics of software production. It impacts the way that 
companies can capitalize on the software they control. 



At the same time, though, open source has no business agenda. Open source is about 
freedom in the political sense. It is about peer review and scientific collaboration. 
Open source licenses take no position whatsoever on the profitability of business 
models. Open source is not antibusiness; it has no opinion. 

Businesspeople, of course, have opinions on open source. Some years ago, when 
open source was still new to the business community, it was unfamiliar, and that 
unfamiliarity bred fear, confusion, and opposition. More recently, as quality open 
source software products have proven their value to all kinds of businesses, that 
opinion has shifted. Most businesspeople who have thought about open source at all 
are guardedly interested in it, and want to find ways to use it in their companies to 
be more efficient, to spend less, or to earn more. 

An informed opinion on open source is important. Just as software is a powerful eco 
nomic force, open source is a powerful force in the development and deployment of 
modern software systems. The Internet, including the World Wide Web, and a wide 
range of the services that run on it (for example, e-commerce sites such as Amazon.com; 
information retrieval services such as Google; and portals such as Yahoo!) exist because 
of open source software. Open source is a genie that is too big for its bottle. Now that it 
is out, it will not fit back in. If you do not put it to work for you, it is likely to wreak bad 
magic on your business. 

My own opinion on open source is simple. It is one tool among many that can, when 
used sensibly, create business value. I run a business based on open source, but my 
agenda is commercial, not political. I understand the politics behind open source, 
and I appreciate and respect many smart people in the open source community. 
When I sit down at my desk, though, I am more interested in the difference between 
income and expenses on my profit and loss statement. To the extent that open source 
helps move that difference in the right direction, I care deeply about it. 

The open source business strategy about which I know the most and the focus of this 
chapter is called dual licensing. Dual licensing is a way to make a single software prod 
uct available under two different licenses. One is an open source license, and encour 
ages sharing and collaboration. The other is a more conventional proprietary license, 
and permits secrecy and competition which promote the creation of proprietary value. 
Dual licensing is a way to give a single product to open source users on open source 
terms, and to paying proprietary customers on conventional proprietary terms. 

Open Source: Distribution Versus Development 

Open source can include a distribution strategy or a development strategy. While 
people often think of the development strategy (many programmers working on a 
common project) when they think of high-profile projects like Linux, we are instead 
going to focus on the distribution strategy. 



72 X Dual Licensing 



Open source is just a way to put product in many users hands inexpensively. Dual- 
licensing businesses do not use collaborative development to build their products. In 
fact, as we will see, that production strategy is poisonous to dual-licensing businesses. 

A dual-licensing business can take advantage of the cheap and ubiquitous Internet to 
distribute its products at low costs. Open source licensing promotes use at much 
lower cost, with much less friction, than an expensive marketing campaign could do. 
Dual-licensing businesses can distribute software to more people, more cheaply, than 
their proprietary competitors. 

At the same time, dual licensing permits these businesses to generate revenue by licens 
ing the software to certain users for a fee. Software licensing revenue is good revenue 
because you can make and license a second copy of a piece of software for essentially 
no additional cost, businesses and the financial markets like licensing revenue. Selling 
support or services, by contrast, imposes new costs with every deal, because a business 
must have the capacity to answer the telephone for every new customer it captures. As 
a result, licensing-based businesses, including dual-licensing businesses, generally get 
higher valuations and can raise capital more cheaply than businesses based exclusively 
on services. 

A Primer on Intellectual Property 

Dual licensing, when you first encounter it, can seem like a parlor trick. How can I 
possibly charge money for software that you can get for free? 

There is, in fact, no sleight of hand. Nobody gets tricked. Dual licensing requires 
some care, and the same diligence that most businesspeople bring to their jobs, but it 
turns out that governments around the world want businesses to be able to do this. 
They have constructed legal infrastructures that permit all sorts of businesses to 
make money with confidence. All it takes is to behave like a business and comply 
with the law. 

Understanding the mechanism here is easier with some background in intellectual 
property law. Most software vendors make money by charging fees for an intangible 
product algorithms and their implementations as expressed in computer code. All 
these businesses, including dual-licensing businesses, exist because of crucial legal con 
cepts regarding intellectual property concepts such as ownership and license grants. 

Ownership 

Ownership of real property is easy to understand my ball, your house, her island. 
Ownership of something intangible like the expression of ideas, is harder to under 
stand. Over several centuries, governments have created rules about who may and 
may not make copies of artifacts like books and music. This "copyright" law bal 
ances the interests of the creator of a work an author or a composer with the 



A Primer on Intellectual Property X 73 



interests of consumers who purchase those copies. Copyright law clarifies issues and 
resolves disputes among authors and readers. Copyright law has been adapted to 
apply to software, balancing the rights of software authors with those people who 
license and use their computer programs. 1 

Copyright law worldwide generally says that the creator of an expression of an idea 
owns that expression. From the point of view of a computer programmer, this means 
that whoever writes a program owns the program. If the programmer copied parts of 
it from somewhere else, of course, he was not the creator of those parts, and he does 
not own them. 

Unless the owner expressly assigns his rights to someone else, he owns his creation. 
Most employment agreements include just such an assignment provision, so the code 
a programmer writes for his job instantly becomes the property of his employer. The 
Free Software Foundation, an important organization in the free software move 
ment, requires such assignment for any contribution to projects that it manages. 

Without an explicit assignment, there can be many different owners of a large soft 
ware product. This is exactly the case with many open source projects, including 
Linux. If 10 developers collaborate on an open source package, each of the 10 owns 
the pieces she produced. None of them owns the entire work. 

Licensing 

Except in very rare circumstances, no one ever buys computer software. The person 
who created the work is its owner; a purchaser only buys certain rights to use the 
copy of the software in his possession. This distinction is critical to businesses that 
build software products. These businesses do not sell ownership. Instead, they sell 
licenses to the software, where "licenses" generally means a collection of rights to 
copy, run, and use the software product. 

The owner is allowed, under the law, broad latitude to set conditions on use of the 
software. If someone wants a license to use the software, the owner can require that 
person to do any number of different things in exchange for the license. 

Most proprietary software vendors require payment of a fee. This has been a remark 
ably successful business strategy for many decades, producing a number of billion- 
dollar software companies. 

Open source software licensing does not require payment of a fee. The owners of 
open source software do impose other conditions on their open source licensees. 



1 The bodies of law dealing with trade secrets and patents are also both important to businesses, 
but are beyond the scope of this discussion. Patents in particular complicate open source 
software development and distribution, and are something of a lightning rod in the debate 
between proponents and opponents of open source. 



74 xx Dual Licensing 



Those conditions vary, depending on the open source license in use, but two broad 
categories are most common: 

Reciprocal licenses require that any recipient promise to contribute back any 
changes or additions to the software. Reciprocal licenses are coercive; they essen 
tially enforce sharing. The most common reciprocal license in use is the GNU 
General Public License, or GPL. 

Academic licenses usually require very little often, just acknowledgment of the 
original owner s work on the software. Academic licenses encourage reuse of 
other programmers work by making it available on liberal terms. The most pop 
ular academic license in use is the Berkeley Software Distribution, or BSD, 
license. 

In both the proprietary and open source cases, the owner of the software sets the 
conditions that others must meet to use the software. 

The owner is allowed to set different conditions for different people. This same situa 
tion applies to real property: you might allow your brother to sleep in your spare 
bedroom for free, but probably would not let me do that on the same terms. At best, 
I could hope to pay you rent for use of the room. Likewise, the owner of a piece of 
software can grant different rights, on different terms, to different people. 

This is common in proprietary software businesses. For example, if you buy a com 
puter with an operating system and a word processor installed, you are most likely 
permitted to use those packages, but not to make copies of them for others to use. By 
contrast, the computer manufacturer is allowed to copy, install, and distribute both 
pieces of software to you and to others. The company that owns the operating sys 
tem and word processor chooses to grant the computer manufacturer broader rights 
than you have, because the owner s goal is to make money in business, and creating a 
distribution channel that generates license fees is a good way to do that. 

The fundamental point to remember is that open source licensing rests on the same 
foundation that proprietary software businesses use for their own licensing: copy 
right law and the notions of ownership and licensing that inform it. In both the pro 
prietary and open source cases, the owner grants licenses to use the software under 
certain conditions. 

One main benefit of being an owner is having the right to set those conditions. 

Dual Licensing 

Dual licensing exploits these attributes of copyright law to construct profitable busi 
nesses using open source software. The software s owner distributes a single product 
under both an open source license and a proprietary license. Open source licensees 
pay no fees, but make certain promises. Proprietary licensees have different rights, 
and pay a fee for that consideration. 

Dual Licensing * 75 



By making the software available on open source terms, the owner creates a large and 
inexpensive distribution channel. Generally, users can download the software on the 
Internet for free. In many cases, the software is bundled with third-party offerings, 
and is distributed by others not formally affiliated with the owner. Open source dis 
tribution makes the software ubiquitous, putting it in the hands of many users at 
very low cost. 

Reciprocity 

Open source, as it has emerged from the academic world, emphasizes the importance 
of reciprocity. Fundamentally, everyone in the community is allowed to share the bene 
fits of the code that others have written. Everyone is expected to share their work, how 
ever. If you give me your work for free, I reciprocate and give you my work on the 
same terms. This emphasis on reciprocity encourages collaboration and allows the 
community to build better software than any small group could on its own. 

In a dual-licensing business, the open source license demands reciprocity. If a cus 
tomer uses the software under the open source license, his own additions which 
may well include the code for his own product must be open source, as well. 

Under the proprietary license, by contrast, reciprocity is not required. The cus 
tomer s own code, and any changes or additions he makes to the original product, 
can remain proprietary. 

This distinction between the two licenses is crucial. Sharing source code is a virtue in 
the open source community, but is a serious competitive disadvantage in many com 
mercial settings. Proprietary software vendors would much rather pay money than 
give away the competitive advantage inherent in their source code. 

Warranty 

A warranty is a promise. These promises are the bedrock of most commercial rela 
tionships. Software vendors that use proprietary licenses to do business with their 
customers and suppliers use warranties to spell out clearly what each party expects 
from the other. 

As a software vendor, I might warrant that my software product will work as docu 
mented, and that I will fix any errors if it does not. I will generally warrant that I 
wrote the software, and that I am the person who owns it. I may ask my customer to 
warrant that he will not give my software away for free to anyone else. If either of us 
breaks one of these promises, our licensing agreement will usually spell out the steps 
that the other party can take to fix the problem. If we cannot fix the problem, either 
of us can sue the other in court for breakingthe promise. 



76 X Dual Licensing 



People do not generally make promises casually. There are real costs in keeping your 
word. For example, if I have promised that my software works, and it does not, I 
need to spend time, and possibly money, to fix it. 

Virtually all open source licenses and certainly all those used by dual-licensing 
businesses are "as is" licenses, with no warranties, no promises, and no recourse in 
the event of problems. The user gets the software as is, and assumes all risk in using 
it. This allows the business to minimize business risk from users of its software who 
did not pay a license fee. Put simply, if the user does not pay for the license, the 
owner will not use his money to protect the user in the case of a problem. 

Proprietary licenses, by contrast, generally include clear warranties. A company can 
afford to make promises to its paying customers, because it can use some of the 
money paid to defray the costs of keeping the promise. 

Depending on how a customer plans to use a software package, he may or may not 
be able to tolerate the "as is" nature of an open source license. If the customer is will 
ing to risk problems in the software, and to fix those problems himself, the open 
source license is fine. On the other hand, if the customer plans to ship the software 
to customers of his own, he may want the assurance of a committed partner behind 
him. Likewise, if the customer runs mission-critical infrastructure on open source 
software, he may need the confidence that commercial warranties provide. 

Competitive Issues 

Many vendors that consider dual-licensing strategies are concerned with competition. 

Naturally, open source distribution presupposes that there are no valuable propri 
etary techniques in the software that need to be kept secret. If an owner distributes a 
software package in source form, competitors can read the source code, just like any 
other person who receives a copy. 

This seems at first like it should be a major concern, but in point of fact, the amount 
of innovation in the software market is much smaller than is generally supposed. 
Especially in relatively mature sectors of technology, such as operating systems and 
databases, the techniques and algorithms are well understood by all the suppliers 
The combination of particular techniques that any single vendor uses may be some 
what interesting to its competitors. All of the vendors in a market hire from the same 
pool of technical talent, and even poach employees from one another. As a result, 
most vendors know pretty well how their competitors products work. 

Proprietary vendors will naturally claim that their products are novel and innova 
tive, and must be kept secret. The problem is that there is no way to test this asser 
tion, since doing so requires examination of the source code, and that is precisely 
what the vendors cannot permit, yet still protect their market positions. 



Dual Licensing X 77 



In some cases, these claims are no doubt true. The strategy of concealment, however, 
does not distinguish between ugliness and beauty, and may be used to hide either. 

A different competitive concern arises from confusion between ownership and licens 
ing. Some vendors considering dual licensing worry that their competitors will 
download the open source version of the product and license it under proprietary 
terms to others. In effect, this would allow a competitor to use the vendor s own 
product to compete with him. 

Copyright law, of course, prohibits this. Only the owner has the right to grant pro 
prietary licenses to the software. In fact, no competitor can copy pieces of the work 
into its own products, since those products would then be forced to comply with the 
terms of the open source license. 

There is another, subtler, force that protects dual-licensing vendors from unscrupu 
lous competition. Open source software is preemptive: the existence of a quality 
open source package makes it very difficult or impossible to introduce a for-fee com 
petitor. The cost of building such a competitive offering has to be recovered some 
how. If a no-cost offering is available, selling an alternative for a fee will likely fail. 
The result is that very few businesses are willing to challenge open source software in 
the market once it is established. 

The converse, of course, is not true. Open source frequently, and aggressively, chal 
lenges established proprietary software. 

Even in cases where proprietary alternatives exist, dual licensing creates competitive 
advantage. The open source product is ubiquitous as a result of an inexpensive distri 
bution channel. In addition, the fact that many users are able to use the software at 
no charge actually takes money away from purely proprietary vendors in the market. 
If a user can choose between a good open source product for free, and a competitive 
proprietary product for a fee, the for-pay competitive vendor is likely to lose the sale. 
Dual-licensing vendors are not paid by their open source users. Neither, however, 
are their competitors. 

Ownership 

Only the owner of a work has the standing to offer it to different users under differ 
ent licenses. As a consequence, dual licensing works only if there is a single, well- 
defined owner of a work. Otherwise, customers have no single entity with whom to 
negotiate their license terms. In practice, projects that use the open source develop 
ment model (a large community of independent programmers collaborating on a 
work) are poor candidates for dual licensing. There are just too many contributors to 
contact for permission every time a new customer wants to license the software 
under non-open source terms. 



78 J* Dual Licensing 



As a result, dual-licensing businesses do not use the open source development model. 
Instead, they invest in the development of the open source software and do not accept 
contributions from the community at large. Dual-licensing businesses rely on open 
source as a distribution strategy, not as a production strategy. 

Ownership is an issue in the larger open source community, of course. As a practical 
matter, assessing the provenance of contributions to an open source project is hard. 
Individual contributors are often judgment-proof, and ensuring that they have not 
misappropriated the intellectual property they contribute to the project requires real 
diligence. Projects manage this problem the same way that proprietary vendors do 
they put experienced managers over coders to review contributions, and they rely on 
mature and experienced contributors to build the product. 

In fact, this ownership issue cuts both ways. Proprietary vendors are equally at risk 
from employees who take pieces of open source software and incorporate it into the 
proprietary products that their employers build. 

The answer for both proprietary and open source developers, of course, is to set up 
formal policies and standards on intellectual property use. Diligence is simply part of 
the job. Both proprietary and open source developers can and do write large, sophis 
ticated products without stealing from others. 

Practical Considerations 

Dual licensing works. Using it, businesses can generate revenue based on license fees 
paid for copies of software. Because the public markets view this revenue as inher 
ently lower margin than services revenue, dual-licensing companies have attractive 
valuations and can raise capital at parity with proprietary software vendors. 

Attractive Margins 

Every sustainable business must consistently earn more money than it spends. This is 
often hard. 

One of the key considerations in designing a new business is the margin that the rev 
enue stream can produce. Margin is, at base, the percentage of revenue that is profit. 
If you pay $100 to build a widget, and you pay a salesperson $100 to sell it, the wid 
get costs you $200. If you charge $400 for it, you have an attractive margin 50%. If 
you charge $205 for it, you have a very thin margin 2.5%. 

Most technology businesses sell licenses, or services, or a mixture of the two. For exam 
ple, a company might license its product to customers, and might also sell consulting 
services to help customers integrate the product into existing IT infrastructures. 

Licensing businesses generally have attractive margins. This is because, while it may 
cost $1 million in payroll to produce a new software program, the second copy of 
that program is essentially free. Having sunk the cost into building the product, the 

Practical Considerations X 78 



company can sell as many copies as it likes without paying the developers to start all 
over again. As a result, selling a new copy of an existing product really just requires 
paying the sales team that sells it. 

Services businesses, by contrast, have much slimmer margins. This is because, to 
deliver the service, the business must have the people on the payroll who can do the 
work that is inherent in the services contract. Selling more consulting engagements 
requires that you hire more consultants. Thus, the costs that a services business incurs 
on every contract include not just the cost of selling it, but also the cost of fulfilling it. 

Of course, very few companies are purely licensing businesses. Most provide at least 
customer support. However, building attractive margins is easier in a business with a 
significant licensing component. 

Dual-licensing businesses offer two different revenue streams. Proprietary customers 
pay licensing fees, giving the business a high-margin licensing revenue stream. Both 
proprietary and open source users may choose to buy consulting or technical sup 
port, giving the business a lower-margin services revenue stream. This diversity in 
the revenue streams can allow the company to weather temporary slowdowns in 
either its licensing or its services businesses. 

Capital 

Over the past several years, the success of open source business models, and of dual- 
licensing models in particular, has caught the attention of the investment commu 
nity. As investors have learned more about open source and dual licensing, they have 
begun to make investments sometimes substantial investments in new busi 
nesses. In addition, established companies have begun to look at open source as a 
strategic business advantage, and not just as a low-cost way to bring technology in- 
house. Major technology vendors have demonstrated their willingness to acquire 
open source businesses at prices that reward the principals and original investors in 
those businesses. 

Venture capitalists and other sources of early-stage funding for companies look for a 
few key attributes in open source businesses. 

First, of course, the company must have a credible, defensible business model. It is, 
generally, easier to raise money for a company that uses dual licensing, than for one 
planning to make money exclusively from services. The big risk for a new services 
business is that it will demonstrate the viability of a new services opportunity, and 
thus attract the attention of a large established services player. Popular open source 
packages have enormous installed bases. Businesses exist today that earn tens of mil 
lions of dollars supporting and training those users. That is enough money to inter 
est even a large, established company. 



80 X Dual Licensing 



The real problem with providing only services for an open source package is that the 
strategy is hard to defend against competitors. While it is true that the original 
authors of an open source package have an advantage in supporting it after all, they 
know the most about how it works freely available source code permits anyone else 
to learn the product internals and offer the same service. 

Investors and acquirers are generally willing to pay less for a pure services business 
than for a licensing business, or for a business that combines the two sources of reve 
nues. After all, costs grow in proportion to revenue growth. Every services sale 
requires staffing to meet the service commitment. Selling licenses for software, on the 
other hand, is much cheaper; making another copy of an existing product is free. 

Second, investors look for open source packages with a solid market presence. The 
investor typically wants to see a track record of successful deployment to prove that a 
market exists. Thus, it is very hard to raise money to build a brand-new open source 
package from the ground up. Indeed, virtually all of the open source businesses 
funded in the past decade have tried to commercialize on the value of existing open 
source packages. 

This creates a chicken-and-egg problem. The open source package must exist before 
it can be funded. Somehow, though, the package must get written in the first place. 
Generally, the open source packages that serve as the foundation for successful busi 
nesses were labors of love in the beginning, created by developers working in their 
spare time for free. Only when they succeeded in building a credible product could 
they raise money. 

Third, and significantly, investors always look at the team in which they are asked to 
invest. As a rule, no investor will back an unbalanced organization all engineering, 
or all marketing, for example. A fundable open source business must combine the 
technical expertise of a solid group of engineers with experienced management. In 
particular, the company must be properly staffed to market and sell the licenses or 
services that the business will offer. This point may seem obvious, but in practice, it 
is very hard to put together a solid team with the ability to make and execute a busi 
ness plan that demands both solid product development and execution against a 
financial plan. 

Once an open source business is successful in the market, it can choose to go in a 
number of directions. 

If the business is generating profits, the owners may choose to continue to run it as a 
privately held company, earning a return on their effort and investment in the form 
of dividends. As a rule, venture-backed businesses do not have this option. The ven 
ture capital community wants to liquidate its investment at a profit within a few 
years of making the initial investment, and will press management to sell the com 
pany at some point. Absent that pressure, however, a strong, profitable, and growing 
business is a wonderful thing to own. 



More commonly, small businesses are acquired by larger businesses, and folded into 
the bigger company. This rewards the original investors, as well as the principals 
who started the company and helped to make it successful. The acquisition will most 
often pay off in cash, stock in the bigger company, or a mixture of the two. 

Established companies are willing to buy open source businesses for a variety of rea 
sons. For example, open source technology is often ubiquitous in the market, and 
controlling that technology can give the big company important advantages over its 
competitors. In addition, an open source platform can create opportunities to build 
new proprietary product offerings that run on top of the open source, which creates 
new service and licensing revenue. The open source business s revenue may, on its 
own, be enough to capture the attention of the larger company. 

The reasons that any particular acquirer chooses to buy any other business depend 
deeply on the peculiar circumstances of each, so there is no cookbook for building a 
company to sell. In general, though, a solid customer base, a track record of consis 
tent growth, and a profitable revenue stream are good things. 

The last way that small, privately held businesses transform themselves is to offer 
their stock for sale on the public markets. This is a much less common practice in 
today than it was six years earlier; a business must, in general, have very high reve 
nues to make this transition. In addition, the demands on the management team in a 
publicly traded company are very different from, and in many ways more onerous 
than, the demands on a private company s team. The requirements for reporting 
financial information to the public markets and the need to manage the public 
investment community s expectations dramatically change the way that a company 
president works. As a result, many companies are forced to change their executive 
teams before offering themselves for sale on the public market. 

Choosing Licenses 

One of the most important decisions in a dual-licensing business is what the terms 
will be for both the proprietary and the open source license. This issue is much big 
ger for the open source license, because that license is never negotiated. People sim 
ply download the software and accept the terms. As a result, a mistake in the open 
source terms is a mistake every single time the software is distributed. 

Academic licenses are poorly suited to dual-licensing businesses, but reciprocal 
licenses generally work well. An academic license simply requires acknowledgment 
of the owner. Most rational businesspeople would much rather make that acknowl 
edgment than hand over precious capital for use of software. A reciprocal license, by 
contrast, requires that the customer s own intellectual property be given away under 
an open source license. There is a very large population of potential customers who 
are more interested in protecting their intellectual property than in saving money. 



82 ** Dual Licensing 



As a result, the only kind of open source license that makes sense for a dual-licensing 
business is a very strong reciprocal license. The best example, of course, is the GNU 
GPL. Choosing the GPL gives you the benefit of the work already done by the Free Soft 
ware Foundation, or FSF, in drafting the license and defining the key terms. In addi 
tion, the FSF has a strong vested interest in demonstrating the enforceability of the GPL, 
so, in the event of a dispute over unauthorized use of your dual-licensed product, you 
have access to a seasoned legal team that knows the subject well 

It is almost certainly a mistake to try to draft your own open source license for a 
dual-licensing business. Doing so requires that you understand and apply the funda 
mental concepts of open source development and distribution in a new legal license. 
This is work you do not need to do if you choose an existing license. More impor 
tantly, writing a new license from scratch creates the opportunity to make bad mis 
takes. Finally, drafting a new license will require you to educate the open source and 
proprietary software business communities about the terms of your own license. This 
will create friction and inhibit adoption. You want developers concentrating on your 
software, not on your license. 

Need and Pain 

The interplay between the software product and its open source license is probably 
the single most important business issue for dual-licensing companies. The software 
product must create need in the market. Ideally, it will be so attractive to customers 
that they simply have to use it. At the same time, the open source license must cause 
enough pain that some users would rather pay money than endure the pain. 

My company makes a product called Berkeley DB. It is an established product with a 
good reputation, but the only way for our customers to use it is to combine it with 
software they write themselves. That act of combination gives us leverage, because 
the resulting work is derived from our software, and we thus can dictate the terms 
under which the derivation must be licensed. 

Our open source license requires that the entire work be released in open source 
form. Open source users, of course, can do this, but the requirement is poisonous to 
most proprietary vendors. Our dual-licensing strategy lets the proprietary customers 
pay for a proprietary license and keep their own intellectual property secret. 

Linux is, once again, a good example of a product for which dual licensing would 
not work. Ignoring the ownership issues raised earlier, no customer needs to create 
and redistribute derivative versions of Linux. Customers simply want to install and 
use the operating system on their computers. None of those actions is forbidden by 
the GPL, so there is no pain. 



Practical Considerations ** 83 



Of course there are businesses making money distributing Linux, but they are doing 
so in different ways. Dual licensing works with only certain sorts of software, with 
very specific ownership characteristics. 

Any vendor considering dual licensing must consider both technology and licensing 
when designing a business model. The software technology must be constructed so 
that users need to do something specific for example, combine it with their own 
intellectual property and distribute the combined work to others to use it. The 
open source license must make this activity painful to at least some customers with 
money. These customers must be willing to pay enough money to avoid that pain to 
make the business profitable. 

On the other hand, the open source license must not be painful to the open source 
community, or it will undermine the benefits of the cheap and ubiquitous distribu 
tion channel that open source licensing provides. Open source users must experi 
ence only pleasure in their use of the software, or the product will fail to penetrate 
the market. 

Measuring the Market 

Businesspeople generally measure markets in terms of dollars the amount of money 
that customers in the market will spend for a product or service. Dual-licensing busi 
nesses need a different metric, because they distribute their software to a combina 
tion of paying and nonpaying customers. The result is that a dual-licensing business 
can have a much larger installed base than a measure of dollars spent would suggest. 

Dual-licensing businesses need to look at this issue pragmatically. Put bluntly, a 
dual- licensing business is never going to get all the money on the table. Some users 
would rather meet the open source terms than pay money for the software. A dual- 
licensing business must balance the size of its installed base, and the concomitant 
opportunity to sell more proprietary licenses, with the money that could be extracted 
by charging for every use. 

The long-term goal of the business is to maximize profits and to grow. By foregoing 
some revenue in the short term, a dual-licensing business may make its product ubiq 
uitous, which creates additional long-term opportunity. Though you might never get 
all the money on the table, you can find yourself sitting at a much bigger table this way. 

There are two other benefits to a large installed base, even if not everyone in it pays a 
license fee. One, noted earlier, is that the earth is scorched for competitors: the only 
way to compete on price with a free product is to pay people to use a competitive 
one. This is an expensive way to capture customers. 

The other, of course, is the opportunity to sell services independently of license fees. 
A large installed base may be willing to pay for consulting and support, even if it is 
not willing to pay for the software. Generally, dual-licensing businesses offer these 



84 X Dual Licensing 



services and book revenue profitably as a result. As a rule, however, dual licensing is 
more valuable for the license fees that proprietary users pay than for the service fees 
that open source users are willing to pay. 

Piracy 

Because open source software is widely available, it is easy for software pirates to 
make and distribute copies in ways that violate the open source license for the code. 
Significantly, it is easy for a user to download the software under an open source 
license, and then to use it in ways that only a paid proprietary license would permit. 

Piracy is not, of course, a problem unique to open source or dual-licensing compa 
nies. Every software vendor indeed, every software developer who distributes a 
product runs the risk that unscrupulous users will pirate copies of the product. 
Individual vendors and umbrella organizations like the Business Software Alliance 
battle the problem continually. 

Piracy is a very real business problem for dual-licensing vendors. The two lines of 
defense are diligence in protecting intellectual property rights, and consistent and 
clear explanations of the terms under which the software is distributed. 

Diligence in protecting intellectual property rights is relatively simple. Anytime a 
dual-licensing vendor learns of a misappropriation of its software, it must pursue and 
resolve the issue. This is no different from the rules that apply to a proprietary soft 
ware vendor, of course. 

Consistent and clear messaging on the license terms is equally important. Open 
source licensing is now generally well understood by the software industry. By 
emphasizing the conditions that apply to open source use, and making clear the dif 
ference between its paid and proprietary licenses, a dual-licensing vendor can edu 
cate its users and capture business before problems crop up. 

In general, no responsible business or consumer wants to misappropriate the intel 
lectual property of another. The penalties are large, the risks are unacceptable, and it 
simply is not fair. Most consumers of software want to obey the law. 

Dual-licensing vendors are in exactly the same position as purely proprietary ven 
dors. Digital distribution means that copying is easy. There are pirated copies of both 
proprietary and dual-licensed software in the world. Vendors of both have the same 
law behind them and the same recourse against pirates. 

The Social Contract 

Open source is much more than an ingredient in a business model, of course. The 
movement predates the dual-licensing model by decades, and its long history has 
produced a complex and nuanced present. Any business that proposes to use open 



source technology, including dual-licensing businesses, must understand and partici 
pate in the open source movement as it exists today. 

The open source community generally cares a great deal about reputation; compa 
nies, as well as individuals, gain status by being smart and by contributing to the 
good of the community. Contribution certainly includes developing open source 
technology for use by others, and any dual-licensing business will do that. Contribu 
tion can also mean, for example, promoting and explaining open source technology 
in the press and at industry meetings a task for which businesses are often better 
suited than individuals and supporting worthy open source projects with staff time 
and money. 

It is unusual in business for an outside group that pays a company no money and that 
has no legislative authority to have as much influence over a business and its policies as 
the general open source community does over open source businesses. If a company 
alienates the open source community, the main advantages of open source distribu 
tion ubiquity and a large installed base can disappear in a flood of bad press and ill 
will on developer discussion lists. Executives at dual-licensing businesses must speak 
clearly to their open source constituencies. They must show the community the respect 
it deserves in order to earn the community s respect in return. 

Besides merely making friends with the open source community, dual-licensing busi 
nesses must pay attention to issues that are unique to the business model. 

An unwritten social contract among open source developers says that the commu 
nity generally will produce software that benefits the community most. This is in 
some sense driven by Darwin the people who write open source code work on the 
problems they consider most important, so if there is widespread pain around a par 
ticular issue, there will be widespread attention to addressing it. 

The social contract, of course, is not perfect. For example, many open source projects 
are more poorly documented than their proprietary competitors, because very few 
programmers enjoy writing documentation, and few will do it without being paid. 
While proprietary enterprise software is not always more polished than open source, 
it is generally the case that the drudgery in proprietary development gets more atten 
tion than it does in open source projects. A programmer working as a volunteer often 
lacks the patience to do the exhaustive testing and debugging, interface cleanup, and 
so on, that commercial licensees of software have come to expect. 

The introduction of companies focused on profits into this mix alters the social con 
tract of open source. The change is, in some respects, for the better, but it is a change 
nonetheless. 

Businesses driven by profits will invest to maximize those profits. A large customer 
willing to pay a significant price for a new feature or for changed behavior in a product 
will get more attention from a business than will a large number of nonpaying users 

86 xx Dual Licensing 



who all want some different feature or behavior. In business, customers vote with their 
money. In pure open source, contributors vote with their programming time. 

A dual-licensing business will, and should, pay attention to the requirements of its 
paying customers. The business needs to balance the needs of its paying customers 
with the needs of the nonpaying open source user community. After all, the com 
pany does not want to alienate its open source users and thereby lose the benefits of 
its open source distribution. 

As a rule, this issue requires attention, but seldom causes real problems. Paying cus 
tomers are usually interested in speed, reliability, or enhancements that make a prod 
uct more powerful. Those improvements are all useful and interesting to open source 
users as well, so it seldom happens that the open source community loses and the 
paying customer wins. 

From the point of view of the paying customer, the ability of money to influence the 
product roadmap is actually a benefit. Many companies considering adopting open 
source technology are concerned that they have no way to influence the develop 
ment community to solve real business problems for them. Because dual-licensing 
companies care about income and profits, customers know that they can get the ven 
dor s attention the old-fashioned way: by pulling out a checkbook. 

Trends and the Future 

Dual licensing is an innovative strategy that combines open source distribution with 
proprietary licensing. The combination confers competitive advantages on busi 
nesses that use it. These advantages include a low-cost distribution channel, power 
ful product marketing, and a high-margin, scalable revenue stream. 

Of course, dual licensing is just one tool that businesspeople can use to build sus 
tainable enterprises in a world where technology and economic forces are in con 
stant change. The rest of this chapter examines dual licensing in that larger context. 

Global Development 

The flow of capital and information across borders has transformed the world from a 
collection of economic islands into a more integrated global economy. That trend 
will surely continue, and will likely accelerate, over the next decade. At the same 
time, security concerns, and particularly worries about terrorism, create a strong 
political incentive to encourage the development of a prosperous middle class with 
an economic stake in the future in emerging economies. 

Worldwide economic development is necessarily influenced by information technol 
ogy. Leaders of emerging nations are clearly interested in building clean, sustainable 
IT industries. Doing so requires a significant investment in education, but also an 
investment in enabling technology. 

Trends and the Future X 87 



Dual licensing provides a way for nascent knowledge economies to grow. Because 
dual-licensed software is available for use at no charge under open source terms, 
emerging businesses can preserve precious capital by choosing to comply with the 
terms of the open source license. 

China is an instructive example. In 2004, the Chinese government announced its 
intention to invest in the development of a version of Linux tailored for use in the 
country, with language and character set support and other new features. The eco 
nomic advantage conferred by a low- or no-cost operating system in a country where 
an enormous number of computers will be installed soon is obvious. Between 2005 
and 2010, China is likely to become the world s largest consumer and producer of 
open source software. Consumption will increase earlier, and faster, than production. 
Demand for high-quality open source and dual-licensed products will be very high. 

Although there is little or no short-term revenue for dual-licensing businesses here, 
they do establish a long-term competitive advantage. As these small businesses in 
emerging economies grow, and as they build value in expertise or new hardware and 
software products, they can choose to pay some of their new capital for the more per 
missive terms of the proprietary license. The cost of switching encourages customers to 
stay with the product that they know, and the one built into their own products. 

This same strategy applies to developed economies. Software vendors are generally 
interested in the education market, in part because they want the next generation of 
software consumers trained to use their products. Many vendors encourage univer 
sity researchers and students to use their products so that the students will prefer 
those products when they graduate and eventually recommend software purchases to 
their new employers. 

Dual-license vendors have an advantage in both cases, relative to proprietary ven 
dors. Because the open source license terms permit use at no charge under reason 
able conditions, individual business owners, as well as researchers and students, can 
choose the open source software easily. They do not need to negotiate a special low- 
or no-cost introductory license to use the product. There is much less friction in the 
distribution of open source software than in most proprietary software distribution, 
and the lower friction translates into higher adoption. 

Open Models 

Dual licensing is, at base, the combination of a venerable business strategy licens 
ing software for money with a relatively new open source distribution strategy. This 
combination is interesting and valuable on its own, but it is by no means the only 
such combination that is possible. 

The global Internet makes the distribution of content much cheaper and easier than 
it ever was before. At the same time, it eliminates barriers to sharing among widely 



88 J* Dual Licensing 



dispersed individuals and companies. The effect is to create new opportunities for 
the collaborative creation of intellectual property. 

Some examples of this collaborative development were all but unknown just a few 
years ago. Weblogs, or blogs, are common today. They have emerged as an alternative 
source for news and information, replacing older media such as newspapers and radio, 
especially for fast-breaking stories. Similarly, the publication and production of scien 
tific journals is changing. Single-vendor control over high-profile journals, and the ven 
dor s ability to dictate pricing to the market, is eroding because researchers can collabo 
rate and publish their research online more easily than ever before. Finally, some web 
sites encourage user contributions to make themselves more valuable to visitors. An 
excellent example is the book and other product reviews on Amazon.com. Amazon s 
visitors share their reviews with one another because they benefit from that sharing, 
but Amazon itself gains an enormous advantage because of the depth and breadth of 
those reviews. 

In all three of these cases, old-fashioned ideas, like news distribution, journal publi 
cation, and product reviews, have been transformed by the open, collaborative pro 
cesses that the Internet encourages. Licensing is as much an issue here as in software 
distribution ownership of the content, and the right to distribute it online, must be 
considered carefully. 

Collaboration and open distribution continue to transform the way that businesses 
operate. That transformation necessarily damages some established businesses, espe 
cially those where a middleman controlled the flow of goods or information, and was 
able to extract a fee from the flow. The Internet allows individuals to bypass that 
middleman the well-known "disintermediation" strategy and to share and pub 
lish on their own. 

Any new business built on a disintermediation strategy must consider the law. 
Unlawful distribution of copyrighted music files is theft, not sharing, but that fact 
alone does not mean that the existing music retailing industry makes sense in a 
world with cheap ubiquitous bandwidth. The next generation of businesses must 
find ways to use the legal system to protect intellectual property, even while they 
take advantage of the powerful collaborative properties of the global Internet. 

The Future of Software 

Open source has transformed the way that software is produced. Now, with dual 
licensing, open source is changing the way that proprietary software is distributed. 

Dual licensing will never replace either pure proprietary or pure open source strate 
gies. There are compelling reasons for both to exist. 



Purely proprietary distribution allows companies to invest significantly in new devel 
opment and to be rewarded for that investment. Particularly at the edge of informa 
tion technology, where the newest products and services are built, businesses will 
use proprietary licensing to protect their competitive position and to extract maxi 
mum value from their efforts. 

Purely open source development and distribution, on the other hand, reduces costs 
of core infrastructure for consumers, and powerful market forces encourage invest 
ment in that cost reduction. Open source has been most successful in those sectors of 
the market where technology is mature and stable operating systems, databases, 
web servers, and middleware. This is not to say that open source development is not 
innovative, but its impact has been largest in the cases where it has commoditized 
products in mature markets. 

The net effect of open source distribution, from the point of view of the consumer, is 
to reduce the total cost of software licensing. Dual licensing does nothing to reverse 
this trend; dual-licensed software will exert downward price pressure on established 
markets just as purely open source software does, though the net reduction in costs 
will be lower since some consumers continue to pay for software licenses. 

Dual licensing will continue to grow in popularity as new and established businesses 
apply the strategy to new opportunities. Competitive pressure ensures that busi 
nesses will look for ways to gain advantages over one another, and building a hybrid 
business offers advantages in many cases. Other novel hybrid strategies will, no 
doubt, appear in the future. 



90 X Dual Licensing 




if CHAPTER 6 

Ian Murdock 




Open Source and the 
Commoditization of Software 



It is said that the only things certain in life are death and taxes. For those of us in the 
IT industry, we can add one more to the list: Commoditization. The question is, how 
do we deal with it, particularly if we are IT vendors and not simply IT consumers, for 
whom Commoditization is an unquestionably positive event? 

Commoditization is something that happens to every successful industry eventu 
ally success attracts attention, and there are always competitors willing to offer 
lower prices to compensate for lesser-known brands or "good enough" quality, as 
well as customers to whom price means more than brand, quality, or anything else 
the high-end providers have to offer. 

Often, to remain competitive at lower price points, the low-end provider employs a 
strategy of imitation for example, investing less in research and development than 
its high-end peers, and instead relying on the high-end providers to "fight it out" and 
establish standards and best practices it can then imitate in its own products. 

This strategy works because success also breeds interoperability. Unless a company 
monopolizes a market (a temporary condition, given today s antitrust laws), an 
industry eventually coalesces around a series of de facto standards that govern how 
competing products work with each other, or how consumers interact with like 
products from different vendors. In other words, given time and a large enough mar 
ket, every industry naturally develops its own lingua franca. 



This kind of natural standardization is good for consumers and for the world as a 
whole. Few people, for example, would know how to type if every typewriter used a 
different layout for its keys, and the telephone wouldn t be in widespread use today if 
each carrier s network couldn t talk to any of its competitors networks. And where 
would we be today without the descendants of typewriters and telephones namely, 
computer keyboards and telecommunications? 

Of course, from any incumbent s point of view, an ideal world would allow, say, the 
market leader in typewriters to own the layout of its product s keys, so anyone who 
learned to type using its product would face huge barriers to switching to a competitor s 
product. Fortunately, the layout of a typewriter s keys and similar interoperability fea 
tures are very difficult proprietary positions to enforce, so once a standard way of inter- 
operating emerges, all vendors are free to imitate that standard in their own products. 

The moral of the story is that standardization, and thus commoditization, are both 
natural market forces as well as key events in human history. When an industry 
matures and competing products become more or less interchangeable commodities, 
this allows new industries to build atop them to create new and innovative products 
that would not have otherwise been possible if the industries they built upon had not 
standardized. In the case of typewriters and telephones, it is clear that the industries 
they enabled the computer industry, e-commerce, etc. greatly exceed the size of 
the industries that enabled them, both economically and in their contribution to 
human progress. 

So, how do incumbent firms fight commoditization? Another moral of the story is 
that they shouldn t. The forces of commoditization, being natural market forces, can 
not be beaten. Yet time and time again, incumbent firms fight them. First, the chal 
lengers are ignored or dismissed as cheap knockoffs, unsuitable for any but the least- 
demanding customer. Then they are ridiculed for lacking imagination and innova 
tion. Then, invariably, they are imitated but by this point, it is too late, as the mar 
ket has fundamentally changed, and the incumbent finds itself unable to compete 
because the challengers were built for a commodity market and the incumbent was 
not. In very simple terms, this is Clayton Christensen s Innovator s Dilemma at work. 

This chapter argues that the open source movement is just another commoditization 
event and that, like other commoditization events, it represents a disruptive shift in 
the software industry as well as an opportunity for entrant firms to unseat the estab 
lished firms against seemingly overwhelming odds. That being said, commoditiza 
tion does not equate to certain death to the established firms if they have the vision 
to see beyond the disruptive events that may befall them in the short term and can 
adapt themselves to the new commodity environment. Above all, this chapter aims to 
convey that commoditization is a natural and unstoppable force that is good for 
everyone involved if that force is allowed to develop on its natural course. 



92 * C Open Source and the Commoditization of Software 



Commoditization and the IT Industry 

The computer industry managed to escape the forces of commoditization for the first 
20 years or so of its life a natural occurrence given the industry was young enough 
and small enough that standards had not yet had the opportunity to emerge. In the 
first two decades of the industry, computer manufacturers delivered an end-to-end 
solution to the customer, from the hardware on up through the operating system 
software that ran the hardware to the applications that ran on top of the operating 
system. Every layer of the stack and, most importantly, the interfaces between 
them was proprietary to the computer vendor. As a result, every computer spoke a 
different "language," and it was difficult to get different types of computers to "talk to 
each other" and interoperate. 

Because of these incompatibilities, the initial choice of hardware implicitly tied the 
buyer to an operating system; in turn, the operating system dictated what applica 
tions the buyer would be able to use. Over time, the high cost of computing technol 
ogy made it financially impractical for the buyer to move away from the incumbent 
vendor because previous investments in that vendor s technology would have to be 
discarded. The combination caused users to become "locked in" to a single vendor. 

However, as the industry matured, the dynamic changed. Entrant firms such as 
Apple, Apollo, and Sun saw the opportunity to create products that targeted an 
entirely new class of computing consumer the individual user that could not 
afford the mainframes and minicomputers sold by established firms such as IBM, 
DEC, and Data General. 

By focusing on "good-enough" quality and lower prices, and by tapping into years of 
consumer frustration caused by batch processing, timesharing, and incompatibility 
between proprietary stacks, the new "personal computing" products were received 
enthusiastically and began to appear in offices and dens everywhere. 

The strategies employed by one entrant firm in particular and one established firm in 
particular would forever change the computer industry. The latter, ironically, would 
lead directly to the commoditization of the hardware industry, and the former would 
lead directly to the ongoing commoditization of the software industry. 

On the hardware side, IBM sought to stem the rising tide of Apple by introducing its 
own personal computing product, the IBM PC. Because of internal cost structures 
designed around multimillion-dollar mainframe products as well as an aggressive prod 
uct launch timeline, IBM decided to use off-the-shelf parts for the IBM PC instead of 
following its traditional approach of developing proprietary components in house. 

On the software side, Sun sought to attain a competitive advantage against the pro 
prietary stacks of the mainframe and minicomputer vendors by basing its worksta 
tion products on the Unix operating system. Unix was already hugely popular in aca- 
demia and corporate research labs, so this approach gave Sun instant access to a large 

Commoditization and the IT Industry X 33 



portfolio of compatible applications as well as an enormous user base already famil 
iar with the operating system that shipped on its products. 

In other words, Unix was an open system that is, a system based on open standards. 
Unix variants from different groups (for example, AT&T Unix and BSD Unix, the 
two variants in widespread use in the early 1980s) were largely based on the same 
APIs. Because of this, applications could be easily ported from one version of Unix to 
another, and users familiar with one version of Unix could easily learn to operate a 
different version. 

Decommoditization: The Failure of Open Systems 

The impact of Sun s decision was the first to be felt. Open systems quickly became 
popular because of the compatibility they offered a completely foreign notion at the 
time. Users adopted systems based on open standards because doing so allowed 
them to move freely among products from different vendors, avoiding the lock-in 
common in the proprietary world. Soon, numerous companies including some of 
the mainframe and minicomputer vendors launched Unix-based workstations to 
compete with Sun s, and Unix became big business. 

As the Unix market grew, the competition for customers became fierce. "Compatibil 
ity among products," which helped the Unix vendors win converts from the propri 
etary world, changed from an asset to a liability. In an attempt to imitate the lock-in 
strategies that had served the mainframe vendors so well for so many years, the Unix 
vendors themselves began adding incompatible features to their respective products. 
This ultimately fragmented the market and alienated customers. By the late 1980s, 
Unix was no longer a lingua franca for the workstation market, but a veritable tower 
of Babel. 

Meanwhile, IBM s decision to use off-the-shelf parts in the IBM PC inadvertently cre 
ated the industry s first open hardware platform. It was not long before a new wave 
of entrants, such as Compaq, Dell, and Gateway, realized they could build products 
that were 100% compatible with the IBM PC, thus gaining access to a large base of 
applications and users, much as Sun had done by adopting Unix. On the component 
side, two companies experienced the biggest windfall from IBM s decision: Intel and 
Microsoft. As the clone market emerged, both companies found an entire market to 
sell to, not just a single company a much larger opportunity, even if that single 
company was IBM. 

At this point, the events set in motion by IBM and Sun intersected. As the Unix ven 
dors were competing vigorously with each other through the introduction of propri 
etary extensions to Unix, thereby "decommoditizing" the lowest level of the software 
stack, the fully commoditized PC waited in the wings. As PCs became more power 
ful, they began to replace workstations, and as PCs continued their march upmarket, 



94 X Open Source and the Commoditization of Software 



the market power of the PC vendors (and, thus, the vendors of their constituent 
components) increased dramatically. In particular, the new ubiquity of the PC helped 
Microsoft s Windows operating system replace Unix as the lingua franca of not just 
the new PC-based workstation market, but also of the entire computer industry. 

Why did Unix fail while the PC has succeeded beyond anyone s wildest expecta 
tions, particularly those of its progenitor, IBM? Both began life as open systems as 
ecosystems of sorts and both grew enormously popular because of their open 
nature. On the Unix side, though, each vendor tried to own the ecosystem by itself, 
and, in the end, all they collectively managed to do was destroy it. Meanwhile, on the 
PC side, the ecosystem won out in the end, for the betterment of all who embraced 
that ecosystem; and, most importantly, the existence of that ecosystem enabled the 
creation of other ecosystems above it. For example, without a truly open platform in 
every office and den, the Internet would not have been able to take root, and it too 
evolved into an ecosystem that has spawned countless products, services, industries, 
and ecosystems that were previously unimaginable. 

Linux: A Response from the Trenches 

It was into this environment that Linux emerged in the early 1990s. At first the mere 
hobby project of a young college student, Linux captured the imagination of those who 
could best be described as the "collateral damage" of the Unix wars. Two features of 
Linux made it appeal to this large group of users and developers: its compatibility with 
Unix, with which they were intimately familiar; and that it was licensed under the GNU 
General Public License (GPL), which not only allowed the scores of Unix refugees to 
contribute to its development, but also guaranteed that Unix-style fragmentation could 
never happen to the result of the community s work, at least at the source-code level. 

Linux grew by leaps and bounds during the 1990s. As with previous challengers, it 
was first ignored, then ridiculed, by the incumbents, primarily Microsoft, which had 
masterfully used its position as the de facto standard operating system to expand into 
numerous additional markets and gain additional even unprecedented market 
power. Unlike so many companies that had come before it, Microsoft wielded the 
forces of commoditization expertly. By offering its products at lower prices than its 
competitors could afford to offer them, Microsoft preemptively commoditized many 
of the markets in which it competed, depending on high volume to make its prod 
ucts profitable and making it impossible for challengers to undercut it. 

As Microsoft s power grew, so did the desire of Microsoft s competitors to counter it. 
By the late 1990s, it was clear that Linux was a powerful force, and many of the 
industry s largest companies began to see it as a competitive weapon. These compa 
nies also recognized that the power behind Linux wasn t so much its technology as 
its licensing and development model, by now referred to as "open source" and in 
particular, the open source model s ability to "out-commoditize" Microsoft. 

Linux: A Response from the Trenches * 95 



The fundamental question is this: why is Linux (and the open source movement it 
helped launch) able to out-commoditize Microsoft? Because it, like the PC, the Inter 
net, and the other open systems and open standards we take for granted today, is 
more of an ecosystem than a technology. Indeed, Linux builds above those previous 
ecosystems without open, commoditized hardware, and without the Internet to 
enable the open source development model to work, Linux would not exist today. 

Microsoft may wield the forces of commoditization more expertly than any company 
that has come before it, but its platform is not an ecosystem. By definition, an ecosys 
tem is an environment to be shared, not owned. Linux is positioned to become the 
lingua franca of the lowest level of the software stack, if we never forget it is an eco 
system and not a product to be owned. Looking at the lessons of the past, if it 
remains an ecosystem, we all win. If not, we destroy it. 

"So, How Do You Make Money from Free Software?" 

If the open source movement represents the commoditization of software, how can 
the challengers of today s software industry utilize its commoditizing power to unseat 
the incumbents, Microsoft in particular? Perhaps more importantly, if this strategy 
succeeds, is there money to be made in a software industry that has been commod 
itized? Finally, are there lessons that can be applied from past commoditization 
events, particularly the events that reshaped the hardware industry in the 1980s? 

For a textbook example of how to turn the commoditization of an industry into busi 
ness advantage, one need look no further than Dell Computer. Dell, of course, was 
one of the companies that started life in the mid-1980s to build IBM clones. Dell s 
initial claim to fame was "build to order," taking advantage of the fact that a PC was 
not really a product in itself, but rather, an assemblage of numerous products that 
any individual with a moderate amount of skill could assemble himself a direct lin 
eage from IBM s decision to base the original IBM PC on off-the-shelf parts. 

Unlike some of its competitors, Dell saw itself for what it truly was: an assembler of 
off-the-shelf components and a distributor of these components in a form its custom 
ers found useful namely, a complete PC. Dell gave its customers choice not an 
overwhelming amount of choice, but enough choice to give those with the skill to 
build their own PCs reason to buy from Dell instead of building themselves. Its com 
petitors, on the other hand, attempted to mold the PC into a monolithic, unchange 
able product, a collection of specific components from specific vendors with the 
occasional bit of proprietary technology added to the mix a thinly veiled attempt to 
decommoditize the PC standard and own it all to themselves. 

To accommodate its new approach to selling hardware, Dell had to develop a new kind 
of business model. Over the years, the Dell model became more about the assembly of 
product than the final product of that assembly process. Dell became remarkably good 



96 * * Open Source and the Commoditization of Software 



at assembling components from a multitude of suppliers into cohesive wholes, and in 
negotiating with those suppliers to get the lowest possible price. It stuck to commodity 
components, allowing the market to pick winning technologies and resisting the temp 
tation to invest heavily in the R&D required to play the proprietary lock-in game its 
competitors were playing. It employed unusual tactics on the sales side, most notably 
selling directly to the consumer instead of going through wholesalers and resellers, 
each of whom took a substantial slice of the profit margin. 

As a result of its streamlined processes and lower cost structure, Dell was able to sell 
PCs at a much lower price than its competitors could. As the PC market grew, and as 
the market commoditized further with each failed proprietary extension to the PC 
standard, Dell s position grew stronger. As the PC began to move upmarket, it sim 
ply became less expensive to "outsource" the assembly of the PCs to a supplier that 
specialized in assembling them, and Dell was extremely well positioned to play this 
new role. Today, as the commoditization of PCs extends to other parts of the hard 
ware market servers, storage, printers, handheld devices Dell continues to be 
extremely well-positioned, and its entry into a new market is often taken as impend 
ing doom for that market s established firms. 

The First Business Models for Linux 

So, what lessons can we leam from Dell as open source commoditizes the software 
world? Namely, that operating in a commodity market calls for entirely different busi 
ness models than the business models that have preceded them. Beyond general conclu 
sions such as this, what specific lessons are there to be learned from Dell s success? As a 
start, we will look at the lowest layer of the software stack, the operating system, and 
attempt to draw parallels between Dell s successful strategy and the strategies of today s 
open source operating system vendors namely, the Linux distribution companies. 

To millions of users around the world, "Linux" is an operating system. They re right, 
of course, but the reality is far more complex than that. First of all, Linux proper is 
just the kernel, or core, of the operating system the rest of the software that com 
prises the "Linux operating system" is developed independently from the kernel, by 
different groups that often have different release schedules, motivations, and goals. 1 

Traditional operating systems are built by cohesive teams, carefully coordinated 
groups of product managers, project managers, and programmers at companies and 
universities. In contrast, Linux is built by thousands of individuals hackers and 
hobbyists and professional programmers some paid to work on specific projects 
but the majority simply working on what interests them. And the reality is even more 
involved: Linux is not just a single system, but hundreds of subsystems, programs, 



1 To avoid confusion, I will use the term Linux to refer to the operating system, following 
standard usage. When referring to just the Linux kernel, I will say "the Linux kernel." 



The First Business Models for Linux * * 97 



and applications, themselves developed by their own communities of individuals 
around the world. 

So, who glues all this mishmash together into something that actually looks like an 
operating system? Since almost the inception of the Linux community, this has been 
the job of the "Linux distribution," a curious term in itself for those coming from 
broader computing circles accustomed to operating systems being built by cohesive 
teams, or at least teams of cohesive teams. 

A Linux distribution is a collection of software (typically free or open source soft 
ware) combined with the Linux kernel to form a complete operating system. The first 
distributions (HJ Lu s boot/root diskettes, MCC Interim) were very small affairs, 
designed simply to help bootstrap the core of a Linux system, on which the user 
(typically a Linux hacker himself, eager to get into writing some code) could com 
pile the rest of the system by hand and as needed. 

A second generation emerged (SLS, Slackware, Debian) that aimed to expand the 
breadth and depth of software shipped by the first-generation distributions, includ 
ing software typical end users of Unix systems might find useful, such as the X Win 
dow System and document formatting systems. In addition, the second-generation 
distributions attempted to be easier to install than the first, as they were targeted not 
at Linux hackers eager to get into writing code, but rather, at the ever-expanding col 
lection of end users Linux was just beginning to attract at the time. 

As Linux s user base grew, many in the Linux community began to sense a business 
opportunity, and the first Linux companies were formed: Red Hat, Caldera, SuSE, 
and many others whose names have long been forgotten. These companies formed 
around the concept of selling commercial distributions to the expanding Linux user 
base. A third generation of Linux distributions was born. 

The commercial opportunity was ripe, as the primary means of acquiring Linux to 
that point had been the Internet, and up to that point, the primary users of Linux 
had been students at universities, where Internet access was plentiful. However, in 
the broader population where Linux was beginning to get noticed, potential Linux 
users were lucky to have dial-up access to online systems such as CompuServe. 
Combined with the rising popularity of CD-ROM drives and the growing size of dis 
tributions to incorporate more and more software to appeal to a wider and wider 
audience, the first business models for Linux were born. 

These business models served the first Linux companies well through most of the 
1990s and, indeed, this is where the term "Linux distribution" originated the com 
panies themselves were little more than assemblers and distributors of Linux soft 
ware, including the Linux kernel, the GNU compiler toolchain, and the other soft 
ware that came with a typical Linux system. As the typical Linux user became less 
and less of a technologist and more and more of a traditional end user, the focus of 



98 * * Open Source and the Commoditization of Software 



the distributions shifted from simple assembly and distribution to making the distri 
butions easier to install and use. 

Linux Commercialization at a Crossroads 

Of course, as distributors of a commodity (for, after all, any company could easily 
become a Linux distributor all the software being distributed was free), these new 
Linux distribution companies lacked the "proprietary advantage" every business 
needs to survive, not to mention thrive. So, following time-honored tradition, many 
of the Linux companies kept their "value add" proprietary in an attempt to better 
compete with each other. 

For a time, one company took a different approach: Red Hat. After a brief flirtation 
with proprietary extensions, Red Hat announced its products would include only 
open source software. Why? It listened to what the market was telling it. The scores 
of Unix refugees, now occupying important positions in the companies that were 
adopting Linux in droves, had already been down that path; furthermore, the giants 
of the industry now supporting Linux, which by now included virtually all of the 
companies that had participated in Unix s destruction and had seen the conse 
quences, saw Linux as a commodity platform that could recapture the position they 
had collectively handed to Microsoft in the early 1990s. As a result, Red Hat emerged 
as the market-leading supplier of Linux software. 

However, as the Linux market continued to grow, and as it began to take a place at 
the core of the computer industry, Red Hat bumped up against its own ceiling, 
caused by lack of proprietary advantage other companies were beginning to take in 
billions of dollars per year in revenue from Linux-based sales, while Red Hat seemed 
to have hit its peak at $100 million or so. 

To counter this, Red Hat came up with a strategy that was still in keeping with its 
"100% open source" market position. Instead of focusing on selling Linux as a boxed 
product, it would sell software updates to those boxed products in the form of 
annual subscriptions. This strategy by itself proved inadequate, as the software 
updates it distributed were available for free. So, it combined the new strategy with 
another maneuver, a redefinition of the "Linux platform" to one it could define and 
control itself. 

Moving away from its traditional, freely redistributable Red Hat Linux product line, it 
launched Red Hat Enterprise Linux. The key part of the strategy behind Enterprise 
Linux was that independent software vendors (ISVs) and independent hardware ven 
dors (IHVs) were directed to certify to this new "high-end" Linux platform, while the 
old Red Hat Linux was relegated to software developers and infrastructure roles. The 
other key pan of the strategy was that Enterprise Linux was no longer freely redis 
tributablethe acquisition of the product was tied to the subscription, and any 
redistribution of the product caused the subscription to be null and void. 



99 



In other words, if Linux users wanted access to the applications and hardware certi 
fied to Red Hat s platform, they had to run Enterprise Linux. To run Enterprise 
Linux, they had to acquire it from Red Hat via the new subscription model, which 
entailed signing a subscription agreement that forbade them from redistributing it. 
More precisely, customers were still free to redistribute Enterprise Linux, but in 
doing so, they lost all support from Red Hat and, most importantly, from the legions 
of ISVs and IHVs that certified to the Red Hat platform. Red Hat s transformation was 
complete when it dropped its Red Hat Linux product line altogether in 2003. Red 
Hat s new model was still in keeping with the letter of the open source movement 
but no longer with its spirit. 

Proprietary Linux? 

By any measure of the term, this is proprietary lock-in, albeit proprietary lock-in that 
does not involve the traditional attainment through source code intellectual property 
i.e., proprietary software. In a way, Red Hat has learned from the lessons of Dell s suc 
cess: it has come up with a clever new business model to match the commodity mar 
ket in which it competes operating systems used to be sold as products in boxes or 
bundled with other products, and Red Hat realized this approach would not be profit 
able in the new operating system market Linux was helping to create; so it found a new 
way to sell its operating system products that was profitable. 

However, in a very real way, Red Hat s model is also dangerously close to the model 
employed by the Unix vendors, which had catastrophic consequences. It is attempt 
ing to decommoditize the Linux platform, not through proprietary extensions in the 
form of software, but through a redefinition of the Linux platform to its own ends 
and the restriction of how that platform can be used and redistributed. Sure, the 
source code to its platform is still freely redistributable, but with the shift of propri 
etary position away from source code intellectual property and to third-party rela 
tionships and subscription agreements, the rules of the game have changed dramati 
cally here as well. 

What s at Stake? 

Red Hat s new business model may be helping its revenues in the short term, but is it 
in Red Hat s best long-term interest not to mention the best interest of the Linux 
ecosystem as a whole if Linux is owned by a single company, or if Linux fragments 
like Unix did as Red Hat s competitors follow down the proprietary Linux path? 

If Red Hat s business model is wrong, what is the right business model for Linux dis 
tribution vendors? In my view, the Dell model can be taken a step further than any of 
the Linux distributors have thought to take it. After all, what are open source tech 
nologies but commodity software components, and what are Linux distributions but 
assemblers of those components into products the end customer finds useful? 

100 * C Open Source and the Commoditization of Software 



Indeed, such an "assembler of commodity software components" business model might 
fully realize many of the benefits of Linux that the traditional, product-oriented busi 
ness models of Linux distribution companies have failed to capture: flexibility and 
choice, without the substantial expertise and financial investment required to adapt a 
Linux distribution for its own purposes. What if a Linux distribution was a collection of 
parts that could be mixed and matched to suit the needs of the company buying it 
instead of a one-size-fits-all, monolithic product like the Linux distributions of today? 

As with new business models that have come before it, such an approach would 
open Linux to new markets, markets that are already using Linux, but for whom 
today s product-oriented business models are ill-suited: server appliance vendors, 
set-top box makers, and others to whom Linux is an invisible vehicle for driving 
their own products. In their world, Linux is a piece of infrastructure, not a product 
to be owned by Red Hat or otherwise. 

Indeed, this is the model being employed by my company, Progeny. Our approach is 
to embrace the commoditizing effect Linux and open-source software have on the 
software industry instead of fighting it. Since every company needs a proprietary 
advantage of some kind, we ve chosen to focus on building advantage through our 
processes, not technology, much as Dell did in other words, to leverage our exper 
tise in distribution building to help other companies assemble commodity software 
components from disparate places into cohesive wholes, and to do so in a scalable 
and flexible way. 

Beyond building a better business model around Linux, what s at stake? I contend far 
more is at stake, for one simple reason: Linux needs to remain a commodity, as it is 
now a core piece of infrastructural technology at the heart of the computer industry. 
Indeed, Linux was enabled by the commodity nature of the last infrastructural tech 
nology to redefine the IT industry: the Internet. 

In "IT Doesn t Matter," which appeared in the May 2003 edition of Harvard Business 
Review, Nicholas Carr points out that infrastructural technologies "[offer] far more 
value when shared than when used in isolation." What happens if Linux is decom- 
moditized and ends up being the proprietary product of a single company to serve its 
own purposes? What if the PC or the Internet had been decommoditized? Where 
would we be today? 

Carr s essay provides hope that there is money to be made in infrastructural technol 
ogies that have been fully commoditized, and that there s no need to try to own those 
infrastructural technologies: 

. . .the picture may not be as bleak as it seems for vendors, at least those with the 
foresight and skill to adapt to the new environment. The importance of infra- 
structural technologies to the day-to-day operations of business means that they 
continue to absorb large amounts of corporate cash long after they have become 



What s at Stake? X 101 



commodities indefinitely, in many cases. Virtually all companies today con 
tinue to spend heavily on electricity and phone service, for example, and many 
manufacturers continue to spend a lot on rail transport. Moreover, the standard 
ized nature of infrastructural technologies often leads to the establishment of 
lucrative monopolies and oligopolies. 

Carr s essay also provides historical perspective on the commoditization process: 

...infrastructural technologies often lead to broader market changes. [...]A com 
pany that sees what s coming can gain a step on myopic rivals. In the mid- 
1800s, when America started to lay down rail lines in earnest, it was already 
possible to transport goods over long distances hundreds of steamships plied 
the country s rivers. Businessmen probably assumed that rail transport would 
essentially follow the steamship model, with some incremental enhancements. 
In fact, the greater speed, capacity, and reach of the railroads fundamentally 
changed the structure of American industry. 

In a commodity world, technologists need to think about innovating in their business 
models as much as (if not more than) innovating in their technology. Of course, it s a 
natural trap for the technologist to think about technology alone, but technology is but a 
small part of the technology business. Look for your competition s Achilles heel, which 
more often than not is an outdated business model in a changing world, not technol 
ogy. To attack your competition with technology alone is to charge the giants head on, 
and this approach is doomed to failure the vast majority of the time. 

Businesses operating in a commodity world also need to build business models with 
the larger ecosystem in mind. It is tempting, once the incumbents have been over 
thrown through the powers of commoditization, to lapse into the same old propri 
etary lock-in strategies that served the former incumbents so well. In effect, though, 
this is decommoditizing the industry, "poisoning the well." It is possible to build a 
successful business in a commodity market, as Dell and many others before it have 
shown, and in the long run, it is far better to ride the forces of commoditization than 
to fight them. 



102 ^ > Open Source and the Commoditization of Software 





t CHAPTER 7 

Matthew N. Asay 




Open Source and the Commodity 
Urge: Disruptive Models for a 
Disruptive Development Process 



Open source hastens software s natural trend toward standardization/commodification. 
While technologically innovative companies will always find ample customer interest, 
the most important innovations for the next decade of software will come from business 
model innovation, mostly spawned by open source license requirements. Open source 
builds a new intellectual property regime centered on the source of code, not source 
code. Protection, in other words, shifts to "owning" the code creator, rather than the 
product she creates. Those business models that acknowledge this and leverage it will 
yield better profits than those that attempt a halfway embrace (or rejection) of open 
source. 

Introduction 

We are missing the point. Yes, open source imposes dramatic changes on the soft 
ware industry, and yes, it is roiling the fortunes of many an established vendor. It 
will continue to do so, and at an increased pace. Yet despite the sometimes 
anguished, sometimes giddy reception that open source has provoked in the IT 
world, open source is not novel. It is not odd. 

Open source is simply the software world s mechanism for becoming just like every 
thing else. 

All the world s a commodity or a service to support and distribute commodities: 
this book that you are reading, the chair that supports you, the restaurant you will 



eat at tonight everything including, increasingly, software, thanks to open source. 
Open source accelerates the natural progress of software toward commodification, or 
standardization. 

It is critical that IT vendors understand this so that they can deploy (or fight, if they 
so choose) open source effectively, and more intelligently choose how and where to 
innovate. Open source does not destroy all value in software innovation; instead, it 
shifts the control point from the code itself to the creator of the code. In so doing, 
open source software will not pillage all closed source software. As in other indus 
tries, there will continue to be plenty of room for upmarket vendors (e.g., Whole 
Foods in grocery retailing; Starbucks in coffee; and Nordstom in retail clothing). 

That said, there is no room for middling and muddling. Open source will commod 
ity from the bottom up while "upmarket" vendors will dominate "up the stack." 
Everything else will be a wasteland. Just as Safeway finds itself pummeled by Wal- 
Mart and Whole Foods so, too, will middle-ground IT vendors find themselves 
grasping at a dwindling market opportunity. 

Open source offers hope, but perhaps not for the reasons normally associated with it. 
Much has been made about the open source revolution, and with good reason. But 
perhaps the best reason has little to do with development of source code, and instead 
has much to do with distribution, marketing, and sales. In other words, what we 
thought was a software development methodology may have far more importance as 
a business strategy that undercuts competitors while driving down costs and shifting 
control to buyers. In such a world, those who understand and leverage open source 
commodification (or escape it) will thrive everyone else will be marginalized into 
economic oblivion. Commodification, the highest stage of capitalism; open source, 
the highest stage of software. 

A Brief History of Software 

Once upon a time, software did not matter hardware did. Software was something 
that hardware vendors wrote to help them sell hardware. Little more. Software was 
important because it made hardware operate, but customers understood that they 
were paying for hardware, and not the software that ran on top of it. (This is still 
somewhat true of certain areas of the embedded software market.) 

As hardware commodified, software grew in importance as a differentiator with these 
same customers. Not all hardware commodified at the same pace: Solaris servers, for 
example, handled a workload in a way that commodified hardware could not, and 
Sun consequently charged a premium. But the real premium increasingly gravitated 
up the IT stack to the applications that people ran on their hardware. Hardware was 
important, but only because the applications had to run on something. With the rise 
of Dell and other commodifiers, however, IT buyers came to care less and less about 



104 x C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



the "guts" of their computing experiences they bought systems for the applications 
they could run, for the productivity they could achieve. 

No single company did more to send the applications trend into hyperdrive than 
Microsoft. Microsoft made software easily consumable by the masses. By simplifying 
computing, and by doing so at a dramatically lower cost, Microsoft grew the market 
by competing against nonconsumption and underconsumption, inviting multitudes 
of average users into the hitherto closed world of computing. Microsoft s Visual Basic 
lowered the bar of expertise to be a proficient developer, and its Office suite created a 
market of home and business users who suddenly could create brochures, quality let 
terhead, etc. Whatever open source developers feelings about Microsoft, they should 
acknowledge Microsoft as the natural parent to their own brand of commodification. 

For this is what open source is doing: commodifying software. We are now at the 
point where mainstream software is becoming commodified by the open source com 
munity, perhaps pushing all value to the services that support hardware and soft 
ware. Microsoft, in a sense, is being out-Microsofted. 

In this world, customers benefit as vendors focus on solving their business prob 
lems, instead of innovating new methods to achieve customer lock-in. Much of 
today s IT world is composed of expensive, monolithic software "solutions" that end 
up creating complexity and integration problems, instead of resolving customer 
problems. That is, today s IT industry is a morass of conflicting standards, complex 
installations, tepid product interoperability, and expense all products of the indus 
try s Wild West "level of thinking" in its adolescent years. Increasingly, however, cus 
tomers are tired of subsidizing the disarray and are turning to open source as a way 
to get more for less. As open source proliferates, the cost of infrastructure software 
will plummet, freeing up resources that the CIO can spend on resolving application 
requirements up the stack. 

Vendors, for their part, also benefit from increased use of open source, because it 
removes the "IP safety blanket." Because code is open, vendors must find innovative 
ways to satisfy and "lock in" customers. Copyright and patent are fine, but they pit 
the vendor in an adversarial relationship with the customer, whereas open source 
control mechanisms tend to force vendors to win by intimately understanding and 
fixing their customers business problems. In addition, this commodification of IT 
will push vendors to move up the stack (and off the stack, into services) to deliver 
increased customer loyalty/value. Finally, as prices for software drop to match the 
drop in hardware costs, more buyers will enter the market, increasing the size of the 
market. Everyone wins. 



A Brief History of Software 



A New Brand of Intellectual Property Protection 

To fully appreciate this trend, it is critical that we better understand the intellectual 
property regime powering the open source revolution. Intellectual property (IP) law 
has always been about control. That control benefits creators by holding off would-be 
competitors long enough to allow the creator to attempt to profit from her innovation. I 
write a piece of software; I copyright it; I sell it (assuming it is a useful piece of soft 
ware and I have adequately marketed it so that people know about it). Simple. This has 
been the software industry s dominant model for decades, and has created a few mam 
moth software companies that have successfully exploited their IP to generate billions 
of dollars in revenues. In this model, exclusion (i.e., the ability to keep competitors or 
customers from copying one s code and distributing it to others) yields profits. In this 
model, the code itself locked up and protected matters most. 

In the open source world, at least as defined by the GNU General Public License 
(GPL), IP continues to play a critical role, but it is a different kind of IP. Dubbed 
"copyleft," open source IP focuses on keeping code access open rather than closed. 
And, unlike in the world of proprietary software, the code matters less than the 
coder anyone can see the code, but not everyone can replicate the coder s influ 
ence on the community to which she contributes her code. By virtue of her contribu 
tion, she builds influence in her chosen code community, and this influence trans 
lates into a new kind of IP: reputation property instead of intellectual property. 

In this new world of open source, reputation property means as much as or more than 
traditional intellectual property. If I employ the developers on a given project, I have a 
measure of control over the direction that the open source project will take. But even 
more importantly, the more developers I employ who work on, say, the PostgreSQL 
database project, the more likely it is that would-be customers will trust me to be able 
to support it. Once a company is thought of as the default support vendor for a given 
project, the harder it becomes to dislodge that vendor. This jibes perfectly with other 
commodity businesses where brand, price, and service provide the only lock-in, a 
benevolent lock-in that customers choose instead of one that vendors impose. 

In this way, the open source code creator exercises a form of control over her cre 
ation, and that control translates into her (and only her) ability to charge a premium 
for the software. As an open source creator, then, my options for deriving profit from 
my creation are not more limited, but they are different. Instead of a limited monop 
oly guarded by law, I have a monopoly guarded by common sense: buyers want to 
buy from the most qualified source of support. They pay to have access to the source: 
not the source code, but the source of the code. 

This distinction is important. The importance of source code gets trumpeted so often 
that one would think that every IT buyer on the planet is clamoring for access to 
source code. They are not. Indeed, Microsoft recently conducted a survey of its cus 
tomers and found that roughly 60% felt that access to source code was "critical." But 

106 * C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



when pressed on the matter, 95% said that they would never look at the source, and 
a whopping 99% said that they would never modify it. (If they did, chances are that 
such modification would violate their support agreement anyway, whether their ven 
dor was Microsoft, Novell-SuSE, Oracle, Red Hat, etc.) In sum, customers perceive 
source code access to be important, but are not exactly sure why. As I will detail 
shortly, the "why" relates to a desire for additional choice and control, and choice 
and control drive down costs. 

Of course, nothing in this chapter should lead one to believe that source code is irrele 
vant. Source matters. It matters because it lowers switching costs (i.e., the cost of swap 
ping out one vendor s software for a competing vendor s software); because it provides 
buyers with more control over their IT, as it allows them to shape code to fit their par 
ticular needs; and because it provides a mechanism for keeping vendors honest, by 
forcing them to take responsibility for the quality of their code. In short, source code 
matters because it shifts control back to the buyer, which forces vendors to offer better 
software at lower prices. While none of these source code benefits requires the inter 
vention of a vendor, we should not get sucked into the belief that vendors matter but 
little in the open source world. Instead, open source actually makes vendors more rele 
vant to customers than ever before at dramatically lower prices. 

Besides benefiting customers, this GPL licensing scheme offers vendors a way to 
exercise an incredible amount of control over competitors. By open sourcing my 
code under the GPL, I push my competitors to follow suit or to increase their R&D 
efforts to escape commodification. Unfortunately for them, this counterstrategy of a 
unilateral R&D arms race tends toward paltry results: customers will often opt for 
the "good enough" product when the price is dramatically lower. Yes, my closed 
source competitors could simply take my freely accessible source code, "fork" it, and 
build it into their own products, but they almost never will. Doing so compels them 
to open source their own software, which they will be disinclined to do. Even if they 
did so, however, and even if my competitor were not a stodgy old closed source ven 
dor, but rather, an agile open source predator, it would matter little, because open 
source buyers invariably favor the source of the source, as it were: they trust the cre 
ator of the code to support it best. 

Open Distribution, Not Source 

This has huge implications for the software industry. Disruptive vendors can opt to 
completely open source their code, relying on reputation property to net them reve 
nues, and further relying on their freely available alternative to competitive products to 
force competitors to meet them on their home turf. This is not to say that all vendors 
must adopt an open source strategy, but rather, that they must compete with open 
source s lower cost structures and superior distribution mechanisms. All must increas 
ingly compete on open source s terms. More detail is needed on why this is so. 



Open Distribution, Not Source ** 107 



The Open Source Weapon 

Open source enables a vendor to maximize its market penetration at minimal cost, 
which is the goal of every IT vendor, but particularly of emerging-growth vendors 
seeking to displace incumbent vendors. One of the biggest roadblocks to any com 
pany s growth is the Bureaucracy Bottleneck the larger the buyer (and, hence, the 
larger the opportunity), the more layers of bureaucracy an IT buyer must fight 
through to try-before-they-buy. Not so with open source, which surreptitiously 
makes its way into enterprises via free download. 

Such distribution fattens a vendor s bottom line without fattening the customer s price 
tag. MySQL had 10 million downloads in 2003, and by mid-2004 had more than 5 mil 
lion installations. Of these would-be customers, 5,000 have returned to buy a support 
contract/license from MySQL, bumping the company s revenues by 100% to $10 mil 
lion in 2003. The revenue growth is important, but even more so is that it achieved this 
growth by spending less than 10% of total revenues on sales and marketing activities. By 
contrast, most public software companies spend 45-50% of total revenues on sales and 
marketing, and companies of MySQL s size generally spend 21.8% on these activities, 
according to a study done by Softletter.com. In short, open source creates a small uni 
verse of prequalified buyers who seek out the vendor, instead of the other way around, 
with the vendor s primary marketing costs relating to setting up an FTP server and 
mostly word-of-mouth-type evangelism to developers. 

The savings do not stop there. Whether the open source vendor "borrows" much of its 
code (e.g., Novell, Specifix, Gluecode) or creates it almost entirely in-house and then 
open sources it (MySQL, SugarCRM, JBoss), open source delivers development-related 
cost savings. For the "borrowers," the cost savings are obvious: they leverage a well- 
developed body of code, most of it written by individuals not on their payroll. For the 
JBosses and MySQLs of the world, which do 85100% of their own development 
work, there is still a significant QA savings from the global pool of testers who submit 
bug fixes and code contributions (which may or may not be used by the vendor) 
cheaper to build, cheaper to sell, cheaper to buy. 

Proliferating Open Source Beyond the Enterprise 

Today, open source largely confines itself to infrastructure software, in part because 
this is where the widest computing community resides. Community-based open 
source projects require a sufficient body of developers with aptitude and interest in a 
given development problem. But as the IT industry begins to recognize the promise 
of emerging open source business models, community becomes less critical to the 
success of a project. As this happens, no area of the software stack will be exempt 

from open source s influence and intrusion. 

r 

Significantly, open source business models will pave the way for open source to 
conquer the Great Middle Class of IT: the small to medium-size enterprise (SME) 



108 ^ C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



market. Open source, as a disruptive methodology in the Clayton Christensen 
sense, will do more than simply allow startup vendors to compete against estab 
lished vendors in established markets: it will help to create new markets by com 
peting against under-consumption and nonconsumption. The Internet, and open 
source s unparalleled use of it, is at this trend s core. 

In "Does IT Matter?" Nick Carr offers one example of how this worked in another 
industry: chocolate. Milton Hershey, founder of the Hershey Company, noticed a gap 
in the chocolate landscape, a gap the railroad could resolve. Until his observation, 
transportation deficiencies had forced production to stick close to consumption there 
was no quick and refrigerated way to ship chocolate over long distances, creating 
scores of micromarkets for chocolate. With the advent of the railroad, however, Her 
shey conceived the idea of a national chocolate market, built it, and owned it. 

Today, the Internet parallels the railroad infrastructure of Hershey s era. It offers 
independent software vendors (ISVs) a similar opportunity to that which Hershey 
had, yet the majority of ISVs are not capitalizing on it. Yes, some traditional ISVs can 
and do offer their products for download, but this is a makeshift attempt to leverage 
the Internet. Open source is an Internet phenomenon it depends upon the Internet 
and extends the Internet s utility. Open source should disproportionately benefit 
from the Internet s distribution mechanism, provided companies understand this fact 
and act accordingly. As I will show shortly, tomorrow s most successful software ven 
dors will triumph to the extent that they develop models that leverage the Internet as 
a distribution mechanism, and use open source licensing as the rules-based system to 
govern that distribution. 

So, Why Not Freeware? 

Let us assume that all of this is true. Open source is great because it enables upstart 
competitors to undercut established vendors on price while providing their custom 
ers Porsche technology at Pinto pricing. But if open source is so important because it 
allows me to freely distribute my product over the Internet, why is freeware not 
equally disruptive?! Stated another way, if the source code does not matter, and only 
distribution matters, why not just give the software away as freeware and charge 
users who require support? Why offer something (access to source code) that simply 
does not matter? 

The easy way out of this apparent quandary is to allow that while open distribution mat 
ters most, open source code access is also important. But this does not get us very far. I 
will therefore detail the reasons that freeware cannot match open source as a distribu 
tion strategy. Most importantly, I will explain how the two matter most when they inter 
sect, making distribution without access as hollow as access without distribution. 



1 Larry Augustin originally needled me with this question, for which I thank him. 

Open Distribution, Not Source * C 109 



Don t view. Don t modify. What do you do? 

As mentioned earlier, it is an indisputable fact that the vast majority of IT buyers will 
never view or modify source code, even if offered the ability. There are numerous 
reasons for this, but the most compelling one is that customers expect to pay for a 
solution to their problems, and not merely a tool to help them solve their problems. 
(More on this shortly.) No company can afford the time and human resources neces 
sary to resolve all IT problems; therefore, they take "shortcuts" by buying software 
that purports to fix certain problems for them. This applies equally well to closed 
source software and open source software. Most IT buyers just want their software to 
work, they don t want to have to fiddle with it. 

By opting not to view or modify source code, does an IT buyer thereby opt out of 
any and all of the benefits of access to the source code? Absolutely not. Just because 
customers do not choose to exercise their rights to view and modify source code does 
not mean they do not benefit from the right, even when not exercised. On one level, 
the option to view the source code serves as a surrogate for the actual exercise of this 
ability. As an example, because I can review the database code that Sleepycat deliv 
ers to me, it forces Sleepycat to provide a higher-quality product than closed source 
vendors would have to offer. 

ROml Lefkowitz of AT&T Wireless gives a tangible example of how this works. In 
June 2003, ROml related that he had asked his wife to solicit multiple contractors 
bids for a home improvement project. Instead of gathering several bids, however, 
ROml s wife procured only one bid. When he asked her why, she responded that she 
figured the contractor would assume she had collected a number of bids, and so 
would give her his best bid from the start. The option to exercise choice, then, served 
her as a useful surrogate for actual choice. 

In this way, access to source code motivates the code s vendor to provide a superior 
product, knowing that it will be open for all to see. It also functions as a security 
blanket for customers. Hopefully, they will never have to look at the source code. But 
if Vendor X fails to deliver on its promises, or if it goes out of business, that cus 
tomer will have the option (unpleasant though it might be) to have some other ser 
vices firm support the stranded code. Source code access lets buyers rely on their 
vendors... but not too much. Importantly, the more independent the buyer is from 
the vendor, the lower the vendor s prices must be. More on this shortly. 

As IT buyers have grown comfortable with open source projects, another benefit has 
emerged. Initially, it is true that buyers will tend to want to avoid tampering with the 
software they buy. Over time, however, as they grow familiar with a product (closed or 
open), the buyer s developers will want to make tweaks here or there. They begin to 
support themselves, in other words, because calling out for support takes unnecessary 
time (and patience). In a closed source world, however, their ability to tweak the "solu- 



110 * C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



tions" they buy is limited. In an open source world, the Amazons of the world have free 
reign over their IT. Source code access gives customers the ability to experiment with 
and tailor software to their liking, if they choose, and on their own time table. 

Other reasons have been suggested. Developers like to be creative. Like anyone else, 
they prefer not to have self-expression manacled, and they choose to express them 
selves in the code they write and modify. On a mailing list in June 2003, Frank 
Hecker argued that access to source code is critical for developers at the heart of a 
company s IT infrastructure. Such people want to be able to modify code, and so 
must have access to source, because they will need to be able to fix any problems 
they may download into the company. Outside the corporate firewall, Ben Tilly (on 
the same email list) stipulated that while the majority of developers are not involved 
in open source, access to source code is critical to that core of developers who do 
participate. They are the lifeblood of open source communities and are the ones who 
will openly extend assistance to newbies who just want the code to work without 
getting their hands dirty. 

All of which is a verbose way of repeating the earlier point: source matters, even 
where it may not directly matter to the end user. Access to source extends benefits to 
users beyond those chosen few who actually exercise the right to touch source code. 
Source matters because choice matters. Choice matters for a number of reasons, not 
the least of which is that choice drives down prices. And choice is amplified by open 
source, not by freeware, which has its source code closed. 

Open source. Open choice. Open wallet. 

Many successful software vendors would have us believe otherwise. That is, they 
want to sell suites of services that take care of all needs, that reduce complexity, and 
that reduce choice. The primary perpetrator of this strategy is Microsoft, which, as 
Dana Gardner of Yankee Group notes, wants to "make the end user any offer they 
can t refuse to go Windows everywhere" (http://enterprise-wtndows-it.news/actor.com/ 
story.xhtmZ?story_td=22143). One major problem with buying into these monolithic 
visions is that once in, the switching costs to go with another vendor are prohibitive. 

By buying into Microsoft or any other vendor that holds out greatly reduced choice 
as a way to accomplish moderately reduced complexity, a buyer surrenders his IT 
destiny to that vendor. He upgrades when the vendor wants him to. It gets new tech 
nology when the vendor chooses to innovate. (I would argue, and have on several 
occasions, that Microsoft s market dominance has caused it to stagnate in terms of 
innovation. When was the last time Microsoft s Office product significantly improved 
over the last version? And yet the buyer keeps buying, because he finds himself on 
the Microsoft treadmill.) And he pays whatever the vendor demands, because the he 
has no other options. He is a prisoner of the vendor s universe, however expansive 
the vendor pretends that universe to be. 



Open Distribution, Not Source ** 111 



Over time, buyers who condemn themselves to such vendor-controlled realities will 
pay more for their IT, both in hard costs and in opportunity costs. Open source 
offers the opposite vision: maximum freedom to shift among vendors (even while 
staying with the same or similar code base). Open source therefore costs less in the 
short term and, especially, in the long term. 

If we step outside the IT world for a moment, this point will become even clearer. 
My wife and I recently redid our landscaping, including our cement work. Or, 
rather, we wisely chose to hire out the work. True, with a Dummy s Guide to Cement 
(my "source code," as it were), even I might have been able to figure it out and could 
have completed the project satisfactorily. Had I opted to do the work myself, the cost 
would have been X. Because I could have done the work myself for X, my cement 
contractor was only able to charge me 1.5X. Had he bid higher, I would have had 
strong incentive to perform the work myself. I had access to the source, so his pric 
ing power was curtailed. (In the same way, access to a closed binary, as with free 
ware, does not accomplish this same effect of driving costs down.) 

The cement contractor ended up performing shoddy work and walked off with a 
portion of our money, for which he had not completed the associated work. Measur 
ing it out, it will cost us 5-10X to hire a lawyer to compel our cement contractor to 
satisfactorily complete his 1.5X worth of work. I happen to be a lawyer, but not one 
that has ever actually practiced law, so I am stuck paying the lawyer s fees: $5007 
hour to recover $1,500+ in payments owed to me by the contractor. 

Perverse world, you say? Yes, I suppose so, but the point is that the delta between the 
cost of me doing my own cement work and the cost of me doing my own legal work 
is directly proportionate to the skill set involved and the artificial licenses set up by 
the legal profession to keep would-be attorneys in "would-be land." I am effectively 
barred from accessing the "source code" of the legal (and medical, among others) 
profession, which drives up the price that I must pay. 

Again, access to source code, whether in software or cement, offers choice, and choice 
ensures lower prices. It does not matter that most people will never choose to do their 
own cement work, just as it does not matter that most IT buyers will never choose to 
view and modify source code. The important thing is that they could if they were so 
inclined. That "could" is instrumental in dropping prices through the floor. 

Such lower prices, then, allow the CIO to spend more money on developers who can 
further customize software to meet that specific organization s requirements. Imag 
ine that: IT that works for a customer, rather than against it. That is innovative. 

Such innovation is what open source is all about, and is why it continues to make 
inroads in the enterprise, in embedded devices, and everywhere else. Open source 
brings choice, and choice saves money. Freeware does not engender such choice. 
Meaningful choice is not created by cost-free technology; rather, choice is created by 

112 * * Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



the freedom to manipulate code to one s personal (or corporate) advantage, or to 
have it widely distributed in such a way that one can benefit from others exercise of 
that freedom. Access matters little without distribution, and distribution matters lit 
tle without access. Together, they spell the commencement of a new age of software 
innovation, innovation that benefits vendors and buyers bottom lines. 

Open Source Business Models 

All of this may sound plausible enough, but we now need to trace through some real- 
world business models that vendors use or could use to leverage the benefits of open 
source to drive revenues and boost profits. Before doing so, it is important to note 
that an "open source business model" means more than simply supporting Linux as 
an operating system. In other words, the fact that my CRM system runs on Linux, or 
that my hardware appliance has Linux as its core, does not make me an open source 
company. An open source business model means that a vendor somehow engages 
with the open source community. 

With that said, I also need to stress that whatever the importance of open source, not 
all companies must adopt open source to find success. Open source is the great corn- 
modifier, but there will always be those who successfully evade that commodifica- 
tion. Other industries prove instructive on this point. 

Take retail, for example. Even as low-cost commodifiers devour middle ground in 
this market, profits persist up the stack. Wal-Mart is the 8,000-pound gorilla of corn- 
modification, cannibalizing groceries, clothing, and just about everything else on 
which it puts its hands. Still, for all of Wal-Mart s success in low-end fashion, for 
example, Nordstrom continues to win at the higher-end game. This is more than a 
case of customer snobbery it has to do with an experience that Nordstrom delivers 
(superior customer service) that Wal-Mart is structurally incapable of offering. (This 
same phenomenon exists in coffee why are consumers so willing to shell out $4 for 
a cup of coffee? Because Starbucks has defined a customer experience that tran 
scends Maxwell House at home.) 

Another example is the groceries market, as Charles Fishman highlights in the July 2004 
issue of Fast Company (http://www.fastcompany.com/magazine/84/wholefoods.html). In 
just a few short years, Wal-Mart has become the United States largest grocery chain, yet 
the title of "Most Profitable Grocery Chain" and "Fastest Growing Grocery Chain" eludes 
Wal-Mart (and its European competitor, Aldi). No, those titles go to Whole Foods, with 
remarkable year-over-year growth in the face of a nationwide 2.5% annual compound 
growth rate: 17% (2003), 21% (2002), 21% (2001), 23% (2000), and 14% (1999). 
Whole Foods delivers an upscale grocery experience, offering organic foods and superb 
quality, and Wal-Mart stocks its shelves according to its modus operandi: decent selec 
tion at rock-bottom prices. Both chains have found their respective strategies to be 
highly profitable everyone else has gobbled their dust. Whole Foods has registered 

Open Source Business Models JJ 113 



$188 million in profits over the last several years, and Food Lion cleared only $150 mil 
lion with seven times as many stores and five times Whole Foods revenues. Safeway, for 
its part, lost $1 billion in the same period on even greater revenues. The takeaway? 
There is no room in the middle for undifferentiated players. One either commodifies or 
evades commodification through innovation. Everyone else languishes. 

Both Source (a.k.a. Mixed Source) Model 

So, the first business model is for the technical innovator that refuses to join open 
source commodification at all. But what about those companies that opt for a "both 
source" model, whereby they offer both open source and proprietary software? This 
model has promise and peril, requiring the vendor to walk a fine line between the 
model s divergent business requirements (low-end commodification/standardization 
coupled with high-end specialization). To the extent that a company marries the two, 
it must do so with a clear understanding of open source complements and substi 
tutes to its proprietary product portfolio. 

Both source offers a way to fill in the gaps left by open source, and to charge a pre 
mium for this "service," while still delivering open source software. Such a model 
seems to be ideal for established players that cannot abandon existing customers of 
closed source products, and blanch at the thought of losing existing profit margins. 
Of course, whether a vendor can avert the open source "threat" depends upon 
whether open source has created a viable substitute to its product. If so, head-on 
competition with that open source project is likely futile unless it can move signifi 
cantly upward in the feature set. Even if it can, competing against free and "good 
enough" is exceptionally difficult. 

A both source strategy makes more sense where the vendor can define and contrib 
ute to open source complements. In economic terms, a complement is something 
that completes a whole; in software terms, it is a component of a software solution. 
So, just as French fries may be considered a complement to a hamburger, so, too, is 
Apache a complement to IBM s WebSphere product. Importantly, the more comple 
ments that exist for a given product, the more desirable that product becomes for 
customers, so vendors want as many low-cost, high-quality complements to their 
products as possible. Oracle likes Linux and x86 hardware because it drives down 
the total cost of a customer s database solution... without lowering the cost of Ora 
cle s software. Customers, thus, can buy more Oracle software, which gives Larry 
Ellison more time on his boat. 

So far, this sounds like a reasonable defensive strategy for vendors that want to toe- 
dip into the open source community without getting very wet. But both source also 
allows vendors to take a scorched-earth agenda against their competitors, by skill 
fully choosing to build open source complements to their proprietary software, com 
plements which cut directly at those areas that competitors have chosen to retain as 



114 * C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



proprietary. Of course, such a strategy must bear in mind the distinct possibility that 
the opposing vendor will then choose to open source pieces of its portfolio that 
injure the first vendor, creating a lot more open source software, but not necessarily 
any profits for either company. 

Still, it is an open question whether this is a solid strategy against upstart competi 
tors with lower costs who can undercut a proprietary product s margins. In addition, 
a both source strategy works best for vendors with products that have respectable 
market share. Slapping open source on or around an also-ran product will not sell it. 
Good technology, good service, and good sales/marketing sell products. Open 
source, by itself, is as much a losing strategy as closed source, by itself. 

Open source complements to a market leader s products make that proprietary prod 
uct more valuable by lowering the total cost of the product. Hence, while both 
source may be the easiest step for a closed source company to make, it will help the 
vendor only if its products were already competing well against other proprietary 
products. Again, both source offers no panacea for market losers. The lesson? Com 
panies should adopt the both source strategy when they are on top of their games, 
instead of when they are losing the final sets of their matches. For market losers, a 
better bet is to make the difficult transition into a pure-play open source vendor, as 
defined shortly. 

Professional Open Source (a.k.a. Services) Model 

The dirty little secret of open source is that the term open source community is some 
thing of a misnomer. In general, the actual number of contributors to any given 
project, including the Linux kernel, is tiny. Thus, to "own" an open source project 
requires little outlay of human resources in terms of numbers, though it may require 
a significant amount of time to build reputation capital within a given open source 
community. (Newbies to the Linux kernel, for example, should expect to put in two 
years or more before they can hope to attain "committer" status in the kernel hierar 
chy.) Despite the low number of developers required to corner the market on an 
open source project, the importance of doing so is massive: employing a majority of 
the developers on a given project roughly equates to intellectual property owner 
ship, as explained earlier. 

For this reason, companies that spawn open source projects e.g., JBoss are able to 
completely open source their code without abandoning pricing power. JBoss, for its 
part, employs roughly 85% of the developers who contribute to the JBoss open 
source project. JBoss offers its code under the Lesser General Public License (LGPL), 
which allows users a wide range of action vis-a-vis their code, including the right to 
fork the JBoss project and start JBoss II. 

But no one does that, for reasons already detailed. 



Open Source Business Models . * 115 



Because of the heavy JBoss "ownership" of the committers to the project, the com 
pany does not save a great deal of money on development costs. It functions much 
like any closed source company, except that its development is open for public view 
and consumption. Any appreciable development savings derive from the bug finds/ 
fixes that JBoss receives from its development community. 

Still, the professional open source business model is not really about development 
savings. Rather, it is about maximizing distribution of one s product; getting it 
beyond the purchasing firewall/bureaucracy bottleneck to plant the product in the 
hands of its developer end users so that they can try and then revisit the professional 
open source vendor for support/service contracts. To get approval to use BEA s 
Weblogic or IBM s WebSphere, a developer would need to go through a cumber 
some process. To use JBoss, she simply needs to click "Click here to download." And 
while the developer might choose to support herself through newsgroups or other 
online fora, in production situations she will generally turn to the source of the code 
(in this case, JBoss). This is the classic open source model, though it is only now 
starting to be exploited effectively. 

Dual-License Model 

The dual-license model has been popularized by MySQL, but has been around for 
some time, most notably deployed by Sleepycat and Trolltech. In the case of a dual- 
license vendor, that vendor employs not most, but all, of the developers who contrib 
ute to the code. Because it employs all of the developers, it also owns all of the copy 
rights to its work. Then, as the owner of the copyrights, it is entitled to license its 
software under one or more different licenses. 

However, the fact that it owns the copyrights and employs the developers begs the 
question as to what benefit, if any, it derives from its open source status. The answer, 
as with the professional open source model, lies in distribution strategy. For a dual- 
license vendor, open source is less a matter of development and more a matter of dis 
tribution for open source vendors. Yes, the dual-license vendor derives benefit from 
outside developers who contribute code (though MySQL, for example, tends to 
repurpose/rewrite incoming code to help it better fit its existing code base) and bug 
fixes, but its primary benefit is in the ability to broadcast its product to the world 
with customers benefiting from lower prices and less lock-in. 

Also interesting, though not a benefit touted by the primary adopters of the dual- 
license model (MySQL, Sleepycat, Trolltech, and now SugarCRM), is the fact that the 
dual-license strategy provides the customers with a mechanism to buy their way out of 
the GPL, if consider this desirable. This is of particular benefit in the embedded world 
where, for example, Linksys might receive GPL d code from Broadcom and might want 
a closed source license to that code, so that it will not have to open source the software 
running its routers and access points (a purely hypothetical example, of course...). 



116 * C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



db4objects is promoting its embedded database with precisely this message, one that 
customers appreciate because however much a vendor may prefer the GPL or another 
open source license, the fact remains that it may not always be the best fit for a given 
customer. As such, the dual-license model offers customers a way to pay for the right to 
choose the license under which to receive software. 

ASP Model 

Such are the prominent open source models that are easily recognized as such: open 
source business models deployed by open source companies. But, as Tim O Reilly 
has been telling the industry for years, "open source" is a much bigger tent than we 
may recognize. Tim includes such "infoware" vendors as Google and Amazon in his 
open source tent. The common denominator between the two? Internet infrastruc 
ture powered by open source (Linux and a great deal else), plus an architecture that 
promotes participation that makes the infoware increasingly valuable. 

The enterprise IT industry has also been moving toward a related model for stan 
dard enterprise applications, calling it utility computing, on-demand computing, and 
a range of other names. In this model, IT vendors deliver computing power in a util 
ity fashion: Enterprise Consumer X gets the computing cycles when it needs them, 
instead of buying all of the hardware/software upfront. 

Importantly, customers in this model buy IT (including software) as a service, rather 
than as a standalone product. As such, customers do not really buy software at all 
they buy solutions to their business problems. Whether the "guts" of that solution are 
open or closed source does not matter anymore. Customers simply pay for value, 
delivered as a service: SP (service property) rather than IP (intellectual property). 

This sounds much like software s ASP model, in which software is delivered to the cus 
tomer over the Internet, hosted on a central server by the vendor, with customers pay 
ing for the value they access over the network. A prominent example is Salesforce.com. 
Whether the software underpinning the service is closed or open source becomes irrel 
evant. The requirement to release modifications to a given open source project is trig 
gered by distribution: so long as the code itself is not actually distributed (and only the 
resultant service dictated by the code is), open sourcing of modifications is not 
required, but voluntary. 

Other Models 

The aforementioned business models are the primary models in use today by most 
open source companies. However, some of the most interesting new companies 
employ equally interesting (and innovative) business models, generally altering the 
way open source software is supported. For such companies, the real customer bene 
fit of open source is the availability of source code. But, as noted, major vendors such 
as Novell and Red Hat, which tie their support contracts to specific product builds, 



Open Source Business Models X 117 



obviate this benefit. Gluecode and Specifix resolve this irony and may point to 
tomorrow s most successful open source business models. 

Managed source model 

Gluecode incorporates three levels of source code support into its model. First, Glue- 
code conglomerates various Apache packages, tests them, and generally makes them 
play nicely together so that customers need not worry about visiting Apache.com for 
themselves. Second, Gluecode develops its own proprietary software to extend 
Apache s reach and thereby provide a compelling solution for portals/business pro 
cess management (BPM). Third, and unique to Gluecode, the company offers source- 
level support to its customers, allowing them to check in their code to Gluecode s 
CVS repository. Gluecode runs the customer s code against test suites to ensure the 
customer s modifications or additions work properly with Gluecode s open+closed 
codebase, enabling the customer to become something more: a development partner. 

Code-level service model 

Specifix does something similar. Focusing on embedded and server deployments of 
Linux, Specifix allows its customers to modify Specifix s Linux distribution to meet 
their particular requirements. Instead of invalidating their support agreement with 
Specifix by doing so, Specifix tracks exactly where the modifications were made and 
allows the customer to support its own modifications, while continuing to support 
the original Specifix distribution. The customer may, in other words, opt for the 
"road less traveled," but Specifix is happy to maintain the more-trafficked road for 
them, keeping it parallel with the customer s chosen divergence. 

Conclusion 

Open source propels software toward Commodity Land, a happy place where cus 
tomers pay for real value and vendors compete on that value, not intellectual prop 
erty lock-in. Each of the open source business models detailed in this chapter will 
help to further this trend, making open source mainstream and possibly displacing 
the traditional, IP-based model as the default. 

We are thus on the cusp of a Kuhnian paradigm shift, one that will fundamentally 
alter the way IT vendors create, sell, and distribute software. Once apparently sty 
mied by the restrictions that open source licensing places on traditional business 
practices by IT vendors, open source vendors are now finding that open source 
licensing creates as many opportunities as it closes, changing the nature of software 
competition for decades to come. This means that incumbent and emerging IT ven 
dors must understand the new rules of engagement to compete effectively. Whether 
they like to admit it or not, open source will force every software vendor to come to 
grips with omnipresent, ravenous commodification. 



118 ^ C Open Source and the Commodity Urge: Disruptive Models for a Disruptive Development Process 



Some may opt for technical innovation over business model innovation, with vary 
ing degrees of success. However, such innovators should recognize that while 
copyright and patent provide potent protections, they also put the vendor in an 
adversarial relationship with the customer. As such, these traditional intellectual 
property tools hurt customers as much as (or more than) they do competitors, and 
will put them at a disadvantage against open source competitors who offer custom 
ers choice and value at lower prices. Open source, then, allows vendors to lay 
waste to their competitors profit margins by lowering their own costs of distribu 
tion, sales, marketing, and development, while simultaneously blessing their cus 
tomers with increased IT flexibility and a more finely tailored approach to solving 
their business problems. 

Open source, then, offers a new way to innovate, a new way to compete, and a new 
way to win. 



Conclusion ** 119 





kf CHAPTER 

Stephen R. Wall! 




Under the Hood: Open Source 
and Open Standards Business 
Models in Context 



People debate regularly whether open source software is "good for business," and 
how one makes money on something given away "for free." People raise concerns 
over the commoditization effects of open source, 1 and portray a gloomy road ahead 
where open source software will "eat its way" up a stack of functionality to the logi 
cal conclusion where software has become valueless. 

Standards as a commoditization driver have been well understood for quite some time 
across many industries. A standard exists to enable multiple implementations. The eco 
nomic argument is that they serve to broaden the market for all producers while foster 
ing price competition (which also fosters production efficiency) for the benefit of con 
sumers. Industry associations of vendors support such work where it expands their 
market opportunities in complementing areas. Governments support such work 
because of the "good" economic effects. Seldom does one hear complaints about this 
commoditization effect, and vendors continue to participate in the development of 
standards and compete on implementations regardless of that effect. 

In this chapter, we will take a look at traditional working definitions of open stan 
dards and open source software focusing on the veneer of differences, then step back 



1 "Will Open Source Middleware Commoditize J2EE?", Nov. 5, 2004, http://linuxworld.com/story/ 
46957.htm; visited Nov. 9, 2004. Also: Paula Rooney, "Open Source Will Commoditize Storage, 
Databases and Security," Jan. 20, 2004, http://www.storageptpehne.com/hardware/J7500067; 
visited Nov. 9, 2004. 



and look under the hood at a broader business context for the dynamics at work to 
provide a business model where standards and open source software can be seen in 
context. 

Open Standards 

A standard can be a specification, a practice, or a reference model. It is used to define 
an interface between two (or more) entities such that they can interact in some pre 
dictable fashion and to ensure certain minimum requirements are met. Standards 
exist to encourage and enable multiple implementations. 

It is important to put some simple perspective on the standards discussions that fol 
low, as books can be written about this seemingly dry subject. We will look at the 
context for standards defined by their development and use, a process for develop 
ing and maintaining standards, and a set of implementation issues such as intellec 
tual property concerns, conformance and certification concerns. Finally, we ll dis 
cuss the history of the concept of "open standards." 

Standardization efforts are typically divided into various categories, but the classifica 
tion systems are often orthogonal. For example: 

Standards can be categorized by the type of development organization e.g., 
national or international body, industry and trade associations, and consortia. 

Standards can be viewed as industry voluntary efforts or government-regulated 
efforts. 

Standards can be thought of as formal de jure developed specifications, or market- 
dominant de facto product technologies. 

All standards live within a context of development and use. Many formal standards 
are developed by national bodies or international organizations such as ISO. These 
standards often define procurement policy for government organizations and large 
enterprises alike. Industry and trade associations develop standards relevant to their 
expert and specialized constituencies. In the information technology space, for exam 
ple, the IEEE has a standards arm, and historically CBEMA (now NCITS) and Ecma 
International acted as standards development organizations in the U.S. and Europe, 
respectively, for IT standards. Each of these three organizations was accredited 
within its national and regional geographies to produce standards that could be later 
adopted by the relevant nationally or internationally sponsored standards organiza 
tion to prevent overlapping efforts, and to build on the relevant expertise within dif 
ferent industry groups. 

Narrowing the focus even further, consortia of vendors often arise within a specific 
area of technology within an industry to develop standards and specifications. The 



122 * C Under the Hood: Open Source and Open Standards Business Models in Context 



consortia often try to build specifications more quickly to expand a particular mar 
ket, feeling that the more traditional organizations are too slow to deliver standards. 

We can categorize standards differently if we bucket them between regulatory versus 
voluntary standards. Government regulation defines a separate set of concerns over 
the voluntary work of many organizations within industries. Such government 
involvement is often driven by economic concerns for the public good (e.g., commu 
nications-related standards) or safety issues (e.g., pharmaceutical testing and registra 
tion requirements or vehicle safety). Regulatory-based standards will not be dis 
cussed further in this chapter because the focus is on the role of standards and open 
source in market-dynamic areas rather than government-regulated areas. 

Another categorization attempts to discuss the difference between de jure standards 
developed in a consensus-based process and de facto standards. A more accurate 
statement might be that de facto technology describes a market-dominant product, 
rather than a specification for interoperability open to all implementers. 2 

Common examples of voluntary information technology standards across this organi 
zational spectrum include SQL, HTML, TCP/IP, and programming language stan 
dards like C/C++ and C#. 

Standards act as a yardstick against which multiple competing implementations can 
be judged in the marketplace to make sure that certain basic requirements are met. 
Vendors compete on implementation beyond the standard to establish competitive 
differentiation in the market. Ultimately, customers choose the product that does 
more than simply meet their base requirements. It is this relationship among specifi 
cation, implementation, and competitive differentiation that provides basic interoper 
ability among vendors, drives competition, and spurs innovation. 

All standards organizations have rules about participation, construction, adoption, 
and amendment. They establish processes for how meetings are carried out to pro 
mote fairness of discourse and prevent anticompetitive practices. Standards develop 
ment organizations also put in place intellectual property rules to ensure partici 
pants are aware of the intellectual property landscape with respect to the standard 
under development. 

Most standards bodies require participating holders of essential patents to announce 
the existence of such patents, and to make them available on "reasonable and non- 
discriminatory" (RAND) 3 terms if an implementation of the standard would require a 
license to the patented technology. Under RAND terms, patent holders cannot dis- 



As we shall see, Clayton Christensen has proposed a situation that says there is market pressure 
that can change de facto technologies into de jure standards. 

While each organization s rules are stated somewhat differently and with different levels of 
formality, a quick look at the governing rules of the IEEE, ISO, IETF, and Ecma International 
shows a remarkable similarity. 



Open Standards *C 123 



criminate against a particular company or a particular platform. Standards organiza 
tions supporting such patent policies ensure that developers interested in delivering 
standards-based products can do so, while ensuring developers that have invested in 
a particular invention still have their investment respected. 

It is important to remember, however, that no standards development organization 
can speak for the intellectual property of developers that are not participants in that 
organization. Standards development organizations structure their patent policies 
this way because they cannot be the policing organizations nor bear the liability for 
patent infringement cases from nonparticipants. They are neither funded nor set up 
to do so. Indeed, if they took on this role, they would likely collapse under the fiscal 
burden and serve no one. 

The interesting thing to observe is that while standards exist to encourage multiple 
implementations, patents are government-enacted legal tools to protect a single 
implementation. Patents exist to allow the developing company government- 
enforced, time-limited legal protection of an invention by preventing others from 
building the invention. It allows the inventing company to recover the costs of bring 
ing an invention to market in return for publishing the idea for future use by the 
broader market. A patent is in some regards the antithesis of a standard. Standards 
are to trade agreements as patents are to tariffs. By definition, they serve different 
purposes in the economic landscape. 

Just as standards organizations are not organized or funded to handle intellectual prop 
erty liability claims, neither are they typically the conformance certifying agencies for 
implementations for the standards they produce. Conformance requirements in the 
standards and specifications are typically simple "claim" style i.e., you provide the 
functionality required by the standard and claim conformance to the standard. Organi 
zations that care about conformance then take on the fiscal and legal responsibility of 
verification around the conformance claims. For example, in the government space in 
the U.S., the National Institute of Standards and Technology (NIST) developed a pro 
curement process (FIPS 4 ) and certification testing process for the standards that it 
cared to use in those procurements. The government was acting appropriately to pro 
tect and serve the public good in federal procurement policy essentially putting pub 
lic tax dollars where its mouth was to improve the return on investment. In a commer 
cial setting, The Open Group (nee X/Open) as a market consortium handled 
conformance claims and liability for its specifications. Beyond the testing requirement, 
warranties of conformance are required and a brand license is signed which is tied to 
the trademark usage associated with the standards it produced. Companies that wanted 
to use the trademark on their products in the market had to pay royalties. The X/Open 
standards were developed through the organization and a company paid for its seat at 



4 FIPS stands for Federal Information Processing Standards. 

124 * C Under the Hood: Open Source and Open Standards Business Models in Context 



the specification setting table through its consortium membership dues. Conformance 
certification, on the other hand, was funded through the cost of trademark use. 

If standards act to define a base functionality to encourage multiple implementa 
tions, essentially the greatest common denominator for a specific technology, they 
help create a commodity. This results in a constant, healthy tension among the stan 
dards bodies participants as they work with each other on the standard, while simul 
taneously vying for market share with their different products. 

The term "open" with respect to standards became a mantra in the late 1980s and 
early 1990s, and was tied to the concept of "open systems." As Cargill observed, 
"open systems" was marketing-speak for the idea that if all the vendors would just 
build their computing products to "open" standards, the consumer would be able to 
build data processing systems by mixing and matching information processing hard 
ware and software modules in much the same way that one could mix and match ste 
reo components to build the desired system. 5 "Open systems" was a description of 
the architecture the consumer thought should exist. Unfortunately, the complexity of 
interconnected data processing systems doesn t lend itself so readily to the metaphor 
of a single-purpose device (i.e., the stereo system) and the ability for plug compatibil 
ity between stereo components to solve all the attendant complexity. 

"Openness" became a quality attributed to the standards that would enable open sys 
tems. The openness was an attribute of the creation process (the standard was built 
in some form of public, consensus-based process open to all participants) rather than 
an attribute of implementations of the standard. 

The development model for a standard is unrelated to the development model used 
for the implementation of that standard. It is equally possible for a standard (open or 
otherwise) to be implemented in a closed proprietary software product or in an open 
source software project. 

Open Source Software 

Open source software (OSS) is a term applied to a collection of software development, 
licensing, and distribution practices. A lot has been written about OSS over the past 
decade, as various open source projects gain market importance and the license models 
demonstrate economic significance. Eric Raymond s original treatise 6 on the develop 
ment practices remains relevant. The Open Source Initiative (http://www.opensource.org) 
publishes the definition of the requirements a license must meet to be considered an 



5 Carl Cargill, Open Systems Standardization: A Business Approach (Upper Saddle River, NJ: 
Prentice Hall 1997), 70-71. 

6 Eric Raymond, The Cathedral &> the Bazaar (Sebastopol, CA: O Reilly Media, 2001). The original 
essay was published in 1997. 



Open Source Software ** 125 



"open source software" license. I will focus on a group of attributes of OSS projects that 
sets up the economic discussion to come. 

OSS projects are interesting "buckets" of technology. Successful OSS projects share a 
number of attributes. 

For instance, distributed communities with good software development practices 
develop technology packages that satisfy well-defined needs. 

Software quality is a measure of community activity (i.e., the developer customers). 

Contributions reflect the individual economic considerations of the contributor 
and are based on selfish asymmetric value propositions. 

The projects reflect their Unix history of loosely coupled component architectures 
with well-defined interfaces that make it easy to assemble larger solutions (e.g., the 
LAMP stack is assembled from Linux, Apache, MySQL, and Perl/Python/PHP). 

OSS projects develop software packages in a distributed community where the core 
developers that inspired the project act as a hub for the evolution of the software as a 
"benevolent dictatorship." Just like all successful software projects, successful OSS 
projects support a strong software engineering discipline and ethic at the project s 
core. Essentially, good software is developed by good software developers. 

What makes the software "open source" is the licensing model. While a wide variety 
of licenses are considered "open source licenses," the basic common denominator 
(without relisting all the requirements from the Open Source Initiative) is that the 
software s source code is always freely available and users can modify it without 
restriction; however, requirements associated with distributing the software may 
exist. In similar fashion to standards efforts supporting a lack of discrimination 
(either in participation within the context of their community or in their intellectual 
property engagement goals), OSS licensing discriminates against no one. Anyone can 
participate in the community development of the software. Anyone is free to use the 
software. Anyone can see the source code. Anyone can distribute the software. In 
each case, requirements may be imposed by the license or reputation that must be 
earned in the community, which would lead some to not want to participate, but 
nothing inherent in the process prevents participation, use, or distribution. 

An interesting dividing line in the licensing schemes is whether the license is consid 
ered "viral." A reciprocal license such as the GNU General Public License (GPL) 
attaches itself to new software by requiring that if the software is modified and dis 
tributed, the license is attached to the new software. This forces the "open" aspect 
upon new software, keeping the source code publicly available. A company may be 
wary of publishing the source code to its software, as it may contain trade secrets or 
other third-party licensed software for which it doesn t have the ability to publish the 
source. The classic permissive licenses arose in academic settings (e.g., the Berkeley 



126 x * Under the Hood: Open Source and Open Standards Business Models in Context 



license and the MIT Project Athena license) and had no requirement to associate new 
work with the license. This class of licenses was very liberal in what was allowed, and 
a company could easily take software, modify it, and not publish the new source 
code. 

One of the most interesting aspects of OSS development is the economics of the 
community participation. Surveys have been run and much has been written about 
the rationale for participation. 7 The "simple" economics is that participants in a com 
munity get more than they give. It is a normal selfish asymmetric value proposition. 
To understand that statement, think about context for a moment. Many people in 
many walks of life use and value their skill sets differently in different contexts. A 
writer might be a technical writer or communications writer for a corporation as her 
paying job, but still use that same collection of writing skills teaching an English as a 
Second Language class in the evenings, working on a writing project with her child s 
class at school, and writing a sonnet to a loved one. In each case, she values her skill 
set differently, and the reward accordingly. Software developers are no different. The 
interesting aspect of community is that corporations are equally economically ratio 
nal in their participation. Developers and corporations participate in OSS projects 
because of the same simple asymmetric value proposition. Many companies partici 
pate in OSS projects and draw upon the software to deliver the products and ser 
vices upon which they base their revenue streams. We will look at this a little more 
closely in a moment. 

Coupling the license and distribution model that ensures the source code is freely 
available, with a core project team that is disciplined allows for the community effect 
of OSS development to shine. The community of interest in a particular project can 
directly contribute changes and bug fixes. While there may be orders of magnitude of 
difference in the number of bug reports submitted, down to the number of bug 
reports submitted with proposed fixes, down to the number of "good" fixes that meet 
the bar defined by the core project team, there is definitely a net gain for the project, 
both from a testing and a bug fixing point of view, as well as the opportunity to find 
new talent for the project that wants to participate. 

The Real Business Model 

Customers view solutions as a network of related "bits" that have to come together in 
some definable fashion to solve their IT problems. This network can be defined with 
nodes representing various technology objects and the paths between nodes represent 
ing the relationships. This is a very informal network, but very real. For example, a 



7 Most notable were the surveys by the Boston Consulting Group (http://www.bcg.com/publications/ 
pub/ication_view.jsp?pubID=935<S language=Eng!ish, Dec. 15, 2004) and the broader FLOSS 
survey done at the University of Maastricht (http://www.in/onomics.nl/FLOSS/report/index. htm, 
Dec. 15, 2004). 



solution for a new retail inventory management system will include nodes representing 
the existing application systems to which the new retail system must interface, com 
puter resources on which it will run, the programming language environment in which 
it will be developed and maintained, the staff and their experience and skill sets that 
will develop and then maintain the new system, databases with which it will need to 
interact, . The other application systems to which the new retail inventory system will 
need to interface have their own historical networks. The platform resources may rep 
resent a different network view if multiple application systems share the fundamental 
computing platform. Companies define architectures for their IT functions to attempt 
to simplify the decisions that need to be made, and often publish these as internal pro 
curement and development standards. History also counts in the network for exam 
ple, some shops always buy "Unix" hardware or always program in C or Java, because 
that is how their resource history has developed. 

Turning the discussion around to the vendor-centric product perspective, Geoff Moore 
defined a model 8 in 1991 for technology adoption that suggests that once a market 
starts to develop, a company best leads by providing a customer the best "whole prod 
uct solution." By this he means that the vendor offers its core value product proposition 
to the customer and then needs to wrap as much around that product as it can to 
present a "complete" product solution to the customer to meet the customer s broader 
needs, essentially mapping as much of the customer solution network as possible. 
Another way to think about this is that the vendor wants to provide as many comple 
ments as it can to its core product offering, covering as much of the customer s solution 
network as is feasible to present the best (most valuable) solution in the customer s eyes. 

The business of a vendor would then be to ensure that the complements were as 
inexpensive as possible, indeed commoditized if possible, so that the whole solution, 
from the customer s perspective, is as inexpensive as possible but the lion s share of 
the revenue would come to the vendor through its core offering. Several business tac 
tics and tools are available to the vendor to try to drive these complement spaces: 

Traditional buy-versus-build strategies can be used to ensure that as much as of 
the customer s solution is provided through the vendor s own brand, regardless 
of whether the complement products are offered as add-ons or are bundled 
directly with the core revenue stream. 

Develop a rich ecosystem of add-ons by encouraging developer and partner net 
works to provide a richer whole solution to the customer. Publishing propri 
etary specifications for the complement space enables more partners to develop 
businesses in the complement spaces. 

Develop tool spaces that help add complements to the complement ecosystem. 



8 Geoffrey Moore, Crossing the Chasm (New York: Harper Collins, 1999). 
128 * C Under the Hood: Open Source and Open Standards Business Models in Context 



Provide certification programs around the core technology to ensure that there 
are lots of service professionals to help the customers complete and support their 
solution. Indeed, a company might have its own consulting services arm for 
parts of a solution, and provide certifications for other parts of a solution. 

Taking this view, a company s assets and offerings also form a network of related 
products and services it matches against the customer s solutions network through 
the sales and marketing functions. Each node in the network has cost, risk, and reve 
nue models associated with it, and as long as the overall revenue model is greater 
than the sum of the costs, the company will be profitable. 

It is important to remember, however, that no company exists alone in the market to 
solve the customer s problems. Each vendor in a particular space must have different 
product networks to allow a differentiation in its sales pitch to the customer. Differ 
ent vendor companies will also behave differently in their hiring and acquiring strate 
gies to shore up their "whole product offerings." 

In addition, it is important to note that one can now look at intellectual property (IP) 
tools (and by that I mean trademarks, patents, copyrights, and trade secrets) in con 
text. Each of these four legal property types or tools (regardless of legal and geo 
graphical jurisdiction) provides a different set of legal protections at different costs. 
One is far more likely to spend heavily and strategically with IP protection tools in 
the spaces defining one s core product value proposition or in spaces in which one 
has the greatest investment, than farther out in the complement spaces of one s prod 
uct offering network. Indeed, in the complement spaces, a vendor may aggressively 
publish (or sparingly strategically patent) to ensure that no other vendor can patent 
in the complement space and raise the prices on that complement. 

If we now stan to consider open source and open standards in this core-complement 
context, we see that they are simply additional tools in the tool chest to drive comple 
ment spaces. Let s look at each separately for a moment. 

Open Source Complements 

It becomes very easy for a vendor (OEM, ISV, or systems integrator) to bootstrap a com 
plement product or project space for its core value proposition to its customers using 
open source software directly. The projects are polished to product readiness either 
within the company or within the community itself. To "buy" versus "build" as comple 
ment strategies for a vendor, we can now add "borrow" and "share." If a vendor joins an 
existing community, it can polish the OSS project to product readiness to complement 
its core value proposition to its customer. If it starts its own project, it can be used as the 
hook to find and engage with new customers around the rest of its core offering. 

The engagement in the community is actually a very leveraged conversation directly 
with people interested in the community s project and then possibly the company s 
offerings. As people cross the line from community participant and software user to 

Open Source Complements * 1Z9 



potential customer, they are self-selecting the vendor s services. This is a very efficient 
way to find new customers. This does not mean one should consider the community as 
a mass-marketing broadcast channel (it s not), but rather, as a public conversation with 
one s customers and potential customers. This is not for the faint hearted. Unlike a tra 
ditional "Go to Market" plan, the technical people have real-time unmanaged discus 
sions with the customers. 9 

The vendor s challenge becomes ensuring that products remain products and com 
munities are communities. Starting a community project is not that risky if the ven 
dor plays by the rules, staffing it with good software developers that will lead the 
community well, and understanding that the real return is the conversation they have 
with customers, and the product complement effect. The "community" at large does 
not exist to work for free improving a company s products. This mistake is still being 
made despite the public experiences of the past. 

The community leadership is a benevolent dictatorship. Sponsoring the community (or 
earning your place in an existing community) does give the vendor the opportunity to 
manage things on its own terms. Software stability is maintained through the commu 
nity project by the leadership. Project direction is developed by the community leader 
ship and people that have joined the community and earned their position of trust. 
There may not be a road map with a view three to five years out, as is almost necessary 
in a product, but the complement space doesn t need the rigor of the core product. 
Viewpoint becomes important. A customer s view of the need for a road map around a 
solution may not map to a vendor s view of the need for a product road map. 

While a number of relatively small companies are using OSS in their businesses, 
large vendor participation is very interesting. [Caveat lector: the following examples are 
observations from the author and do not represent any direct knowledge of these vendors 
business plans or models.] 

IBM has made three big plays: Apache, Linux, and Eclipse. IBM joined the Apache 
community six years ago, borrowing a web server while selling WebSphere. It joined 
the Linux community four years ago while managing the commodity curve on the 
AIX product line and using it as a competitive shot into the Sun server market. Most 
recently, it has begun a "share" project creating the Eclipse project out of technology 
it acquired (and then it acquired Rationale). 

In joining the Apache community, IBM doesn t need to maintain its own web server 
team and can focus its efforts on WebSphere instead. In the Linux community, it can 
focus on the parts of the OS that best meet its needs. Linux is clearly becoming the Unix 
server replacement over time. IBM s AIX product space will be replaced. It can either 



The first thesis in the Cluetrain Manifesto is "Markets are conversations." Indeed, most of the 95 
theses are highly relevant to the discussion (http://www.duetrain.com/fimanifesto, Dec. 15, 2004) 



130 I * Under the Hood: Open Source and Open Standards Business Models in Context 



actively participate and position itself on the leading edge of the curve, or wait until its 
product space is consumed. 

SAP released a complete modern relational database for free in August 2002 to drive 
its core business into the mid-tier customer space where the customer may not 
already have an enterprise-class database and may not be willing to pay the "Oracle/ 
IBM/Microsoft" tax to get SAP R3. It was released under the GNU GPL after a two- 
year, 100-person investment in updating the acquired Adabas technology. SAP then 
partnered with MySQL AB in Sweden to "manage" the database community. 

Sun Microsystems worked in the GNOME desktop community to develop, acquire, 
and contribute the accessibility features it needed to meet U.S. government procure 
ment policies to complement its Linux workstation offerings. For a relatively modest 
investment in the tedious and difficult accessibility technology, it is getting an entire 
full-featured desktop environment. 

In each case, the corporation is getting more than it gives, developing a complement 
rapidly around core offering(s). They gain time-to-market for the complement at a 
reduced investment. While initially met with skepticism when a large company joins 
an existing community, as long as that company plays by the community s rules with 
respect to engagement and quality, it can become as accepted as any other active par 
ticipant. Depending upon the nature of the product relationship to the core and 
company commitment, the company may make best efforts to hire key community 
developers. This is not altruistic, but neither does the company expect the develop 
ers to change their community engagement. It gives the company deeper insight into 
the community it is looking toward for support as it develops the complement. 

There is a competitive edge to OSS community development as well. Often the com 
pany takes advantage of the reciprocal aspect of the licensing to salt the intellectual 
property fields around it by aggressively publishing prior art, holding the comple 
ment costs down, and preventing competitors from directly monetizing their origi 
nal investment in the community project software. For example, SAP is not in the 
database business and so may feel comfortable publishing the investment in SAPDB 
(now MaxDB), but it probably doesn t want Oracle, Microsoft, or IBM directly mak 
ing use of that investment in their respective database products. In this case, the 
reciprocal license is the most business-conservative license SAP could choose. As well 
as driving a complement directly, the community engagement also allows the ven 
dor to work closely with partners, customers, and potential customers to build the 
relationships they will need to sustain the business over time. 

The other competitive aspect happens when you consider two competing vendors 
product-centric networks, and how they appear to the customer. The customer is 
looking at things as a "whole product solution" and does not really think (or care) 
about what is core or complement from the vendors perspectives. A vendor can 
develop a complement community directly in the path of a competitor s core value 

Open Source Complements ** 131 



proposition to a mutual customer. It need not be a deliberate move and the sole pur 
pose of a community; it is the icing on the cake of the multifaceted approach of a 
business in using OSS development and engaging with its customers. 

Small companies can also easily use the OSS buckets to bootstrap product comple 
ments. Clayton Christensen s original research 10 around disruptive business models 
shows how small companies assemble off-the-shelf parts into underperforming prod 
ucts compared to the industry norm, offering those products in their own niches 
with different business models. As the sustained innovation around the new disrup 
tive product develops, it eventually becomes mature against the yardstick used to 
judge the incumbent but at a better price for the performance, and the incumbent s 
business is disrupted. Consider the development of the Linux operating system 
from its inception in 1991, delivered by a university student, its growth in educa 
tional use, to simple infrastructure servers, to the point in history where it is pres 
ently challenging the traditional Unix vendors products (though it has, in some 
cases, become too complex to teach anymore 11 ). 

There is also a situation, as we shall shortly see, where a product market hits the point 
when customers start to be overserved, and there is a call for standardization. This 
means that OSS components that already represent a package with well-defined inter 
faces may be a rapid way to bootstrap a "good-enough" product into that market. 

One thing to note in this discussion using the network of core and complements 
together is that there is no "stack" of technology per se. Think back to the earlier dis 
cussion of customer-centric solution networks and vendor-centric product networks. 
Vendors may see their world as a stack with their valuable core at the top and all the 
commoditized complements below, but in reality, it is simply their view through their 
own product stack and its relationship to the customer and their partners. A chip man 
ufacturer views the stack very differently from an operating system company or from a 
middleware company (hardware design in silicon is where the value is, with operating 
systems and middleware and apps being less and less interesting to the chip manufac 
turer). The terminology of eating up the stack may have more to do with the position 
in which the vendor perceives itself. 

Open Standards Complements 

Clayton Christensen further observed in his research 12 that as companies begin to 
overdeliver functionality in their product lines faster than customers are able to use 
the new functionality and therefore faster than customers are willing to pay for it 



10 Clayton Christensen, The Innovator s Dilemma (New York: Harper Collins, 1997). 

11 In interviews in February 2003, a number of university OS professors made reference to the 
current revision, with the addition of symmetric multiprocessor suppor, becoming too complex 
to teach. As a result, they were basing their course work on earlier versions of Linux. 

12 Clayton Christensen, The Innovator s Solution (New York: Harvard Business School Press, 2003). 

132 ^ C Under the Hood: Open Source and Open Standards Business Models in Context 



the market begins to call for standardization. Indeed, prior to the point where they 
begin to overdeliver, the market leader is often offering the technology in a tightly 
integrated fashion and best delivers to consumer needs in this space where the solu 
tions typically are not yet good enough. This is the time when tight integration, not 
standards-based components, is the path to success. Standards develop once the 
marketplace reaches a point where the market leader begins to overdeliver. These are 
the circumstances in which a market-dominant de facto technology is at a critical 
point and the call for de jure standardization is possible. 

The signal to standardize a technology is somewhat unclear, but there is likely a col 
lection of factors: 

Competitors with standards experience and similar product offering networks 
but different core drivers likely use the opportunity to "call for standards," 13 
hoping to reduce their own complement costs while causing a competitor grief 
in a core revenue stream. 

Customers managing substantial procurement budgets will support and call for 
standards in the hopes of prolonging investments and attempting to reduce costs 
from vendors that are overdelivering. For example, the U.S. government as the 
largest IT buyer on the planet at the time, led the charge around the POSIX and 
C-language standards, quickly followed by the large companies in the petroleum 
and automotive industries. 

If you are the one true implementer, and the market (i.e., partners, customers, and com 
petitors) is calling for standardization in your core technology space, you have a prob 
lem. They re calling for the benefits of standards (expanding market and price competi 
tion) because they want the ability to replace you. Some segment of your customers 
wants the choice of multiple implementations. Your competitors are happy to support 
the call, as this is the thin edge of the wedge to break open your value proposition to 
your customer, all in the name of open systems. Your partners may be happy to support 
the call for standardization because they want price pressure as their margins diminish 
and perhaps your percentage of their Cost-of-Goods-Sold is increasing. 



13 Geoff Moore argues that the first response in the market from competitors when they see a 
"gorilla" forming is to cry for "open systems" (Geoffrey Moore, Living on the Fault Line [New 
York: HarperBusiness 2002], 119). This might be more of a cause for standardization too early 
with all the attendant problems that ensue as has been observed by James Gosling of Sun 
Microsystems (James Gosling, "Phase Relationships in the Standardization Process", circa 
1990). Goslings observations are more closely in line with Christensen s, arguing that there is 
an optimal time in a technology s development for standardization. Some of us have always 
suspected that it is best to standardize existing practice and experience, instead of trying to 
standardize ahead of the market curve. Indeed, it would be interesting to do a survey of 
successful and unsuccessful standardization efforts to determine whether the unsuccessful 
efforts were undertaken too early in a marketplace, when vendors are still trying to define the 
marketplace itself and stake out claims with products and patents. First, of course, one would 
need to define the measure of a successful standard. Christensen s observations are likely more 
in line with standards forming at the optimal market time. 

Open Standards Complements ** 133 



It is important to note that one needs to get the view of the market "right" for this 
sort of discussion, and hindsight is always 20/20. It is not necessarily the dominant 
vendor s product that is to be standardized, but the product market space. For exam 
ple, one can argue that the POSIX standards (and the C-language standards, for that 
matter) were not about standardizing Unix systems, but rather, were an effort to 
standardize an OS interface for minicomputers. Digital Equipment Corp. was the 
dominant player in minicomputers (which became departmental servers and work 
stations). DEC was driving customers up the hardware upgrade cycle to support its 
market growth faster than customers were willing or able to absorb the change. Unix 
systems of the early and mid-1980s represented the best opportunity around which 
the market could form a minicomputer application programming standard to sup 
port customers applications portability. While the Unix systems of the day were 
often less scalable, less robust, and less secure than VAX/VMS systems, the Unix 
operating system had been ported to most vendors hardware (including DEC 
VAXen), so competing vendors could see the market opportunity. 

At the same time, the PC arrived on the scene. Many have argued that the PC won 
against Unix systems by taking over the desktop, largely due to the inability of the Unix 
vendors to set a desktop "standard" fast enough. The PC certainly took the desktop by 
storm, but it was actually competing against nonconsumption. In a Christensen view of 
the world, it was put together from inexpensive parts, and when compared to mini 
computers it was certainly underperforming, but it became the de facto business appli 
ance in a document-centric world, enabling a whole new class of electronic document- 
centric applications. (Word processing systems companies vanished almost as fast as 
the minicomputer companies.) The PC was competing with nonconsumption, giving 
business users computing resources on their desktop instead of being stuck waiting for 
their business data processing applications to be developed by corporate IT, with its 
ever-growing systems development backlog. The Unix systems (driven by standards 
and an "open systems" message) were data processing-centric rather than document- 
centric, and caused DEC grief in a completely different space. 

Christensen observed that as an area of technology is standardized, the value moves 
to adjacent spaces in the network. 14 The trick then becomes to ensure that one is 
building one s business efforts in the product network around the space being stan 
dardized. This would lead us to believe that the richer a product offering network a 
vendor has, among different software, hardware, and service components and prod 
ucts, the more opportunity that vendor has to move with the value or to define new 
components that the old components complement. 



14 This was originally referred to as "the Law of Conservation of Attractive Profits," but is now 
referred to as "the Law of Conservation of Modularity." 



134 X Under the Hood: Open Source and Open Standards Business Models in Context 



This core-complement product network view allows one to very rapidly see how the 
vendor politics in a standards working group play out. A vendor with a de facto 
product technology that is being dragged by the marketplace into a de jure stan 
dards working group is likely a little less than enthusiastic about participating in its 
own commoditization. The vendor alliances within the working groups are partici 
pants in the complement space. The game is one of technology diplomacy, where the 
goal as a vendor representative is to expand your area of economic influence while 
defending sovereign territory. This holds true regardless of whether one is participat 
ing in a vendor-centric organization such as Ecma International, as an "expert" to a 
national delegation to the ISO (on behalf of her employer), or as an individual con 
tributor to an organization like the IEEE (again, funded by her employer to partici 
pate). Vendor consortia offer a similar view. Which vendors formed the consortia 
and which vendors quickly and noisily joined shortly afterward says a lot about who 
the incumbent in a product space is and who the competitors are. 

Conclusion 

Businesses are often much more than simply hardware companies or a software com 
panies or a service providers, offering breadth of product and service in overall value 
proposition to their customers. Successful companies use a collection of strategies to 
deliver a "whole product offering" for their customers, driving their core revenue 
generator with a host of complementing products and services. 

Standards have traditionally been one tactic or tool for driving additional comple 
ment value to a customer by developing a complement space in a maturing market 
with a lot of implementations at a reduced price. 

Open source software can also be used as a tool to develop a complement space that 
supports a core revenue product or service. The open source project can act as a 
quick and convenient bucket of technology around which other product offerings are 
wrapped, or plugged into an existing product offering network. 

A number of models were presented on how to think about customer-centric solu 
tion networks and vendor-centric product networks (and for thinking about the 
product network from a core-complement point of view), alongside Moore s tradi 
tional Technology Adoption Life Cycle and Christensen s models for how product 
markets behave. Open source software projects and standardization efforts can be 
viewed as tools to be used to attain competitive advantage. A number of large corpo 
rations are now participating in OSS communities to the benefit of the corporation 
and the communities, just as corporations have historically driven voluntary stan 
dards engagements. The model-based view certainly doesn t take away from the 
excitement inherent in different OSS projects or the overall economic value of a suc 
cessful standard. It merely provides context to businesses that want to understand 
how to adopt and participate in either. 



Conclusion ** 135 





t CHAPTER 9 

Russ Nelson 





I ve been giving away my software since 1983, full time since 1991. I don t do it for 
fun, although I enjoy it. I do it because it s a way for a small business to earn money 
and it s fun. Each of my software interests started as a hobby, and some have turned 
into a profession. Not every hobby of mine has turned professional, and I hope to 
explain why some have and some have not. 

Three of my hobby projects, which I ll talk about in depth after I introduce myself, 
have turned profitable. They are Freemacs, Packet Drivers, and qmail. 1 Freemacs is an 
MS-DOS text editor, styled after Emacs. It s still used today as the official editor of the 
FreeDOS project. Packet Drivers hide the difference between Ethernet cards in an MS- 
DOS system. If you ve ever eaten at a McDonald s restaurant, your order was communi 
cated through Crynwr Packet Drivers. Qmail is a mail transfer agent (MTA) for send 
ing and receiving Internet mail. Qmail is the engine behind Rediffmail s 30-million- 
user, multiterabyte, 100-node email cluster, and many smaller sites. 

Introduction 

I did hardware hacking long before college. Digital electronics was too expensive for 
me: $1 per TTL quad nand gate at a time when vinyl records cost only $5. So, I fid 
dled around with analog electronics. I invented a trigger sweep for my dual-beam 
oscilloscope, and an analog computer throttle for my model railroad. 



Qmail is an all-lowercase name, and will be capitalized here only at the beginning of a sentence. 



^137 



My high school was a member of LIRICS: Long Island Regional Instructional Com 
puter System, which had a PDF- 10 students could use to leam to program, via tele 
types operating over modems. I was at Baldwin Senior High School from 1972 to 1975, 
but took advantage of the program only in my last year, from 74 to 75. I learned 
BASIC and wrote a four-banger calculator program. I also wrote a word processor in 
BASIC, for which I had to do all sorts of horrific string manipulation. It took hours to 
format a two-page social studies paper. Partway through I got an Instant Message (IM) 
from an operator who asked me what I was running, and if it was looping. 

During this period I learned PDP-10 Assembly language. JRST, HRRLZ, and SKIPNE 
are all familiar friends to me. Unfortunately, none of my candidate colleges had a 
PDP-10. MIT almost certainly did, but I was a poor scholar who was more interested 
in getting an education than in proving that I had one. Of course, everything on the 
PDP-10 was what we d now call "open source." Nobody thought of holding back the 
source code in those days. 

Many colleges will teach you how to become a businessman. I didn t have that desire 
upon entering college, and so I sought a degree in electrical engineering. Only later 
did I decide to run my own business, but how to learn? I started slowly, learned 
through experience, and didn t take too many risks. 

I learned "on the job," and discovered new ways to profit from open source soft 
ware. Everyone in the business had to teach themselves. There is no master s degree 
of open source business administration not yet anyway. 

Hewlett-Packard recognized my "genius" and hired me and my wife to work in its 
calculator division doing integrated circuit design. I missed programming, so I 
bought a RadioShack Color Computer (CoCo). This led to my first freelance income 
associated with programming. 

I wrote programs for fun and sold them to CoCo Magazine for distribution. They were 
just little cute things, but they were in Assembly language, so they were fast, small, 
and easy to distribute. The standard distribution was on audio cassette through a 
paid subscription. It was nice to receive money for writing a program for fun. With 
out a local user community, it was also the only way I could distribute my software. 

Freemacs and Open Source 

After I returned to graduate school, I started writing a programmable editor for MS- 
DOS. Instead of writing it de novo, I thought it should be compatible with Emacs. I 
had used Emacs while working at Hewlett-Packard, where I had done some hack 
ing on an editor. That experience convinced me I wanted an editor without any dis 
tinction between editing and typing. I purchased a copy of the MIT AI Lab memo 
describing Emacs, written by Richard M. Stallman (RMS). I recognized his name 
because of his GNU Manifesto, a new document then. I sent email to RMS to get 



138 K * Open Source and the Small Entrepreneur 



permission to sell a copy of the memo along with my editor. The document was in 
the public domain, but I thought that asking permission was only polite. 

I got a call from RMS (this was back when his wrists hurt so badly that he couldn t 
type). He persuaded me to give away my version of Emacs rather than sell it. He 
appealed to my sense of fairness. He asked why I should profit from a manual that I 
had not written. I was impressed that so stellar a personage as RMS would take the 
time to call me. I decided that it would be best to give it away. I didn t really have 
any idea whom I might sell it to. All my previous software sales had been for the 
RadioShack Color Computer, and had been sold to CoCo Magazine. My editor was 
for MS-DOS, so this was a completely new situation for me. Once I decided to give it 
away, I gave it the catchy name Freemacs. 

In graduate school, I had access to worldwide networks, and it was actually possible 
to distribute software "for free." Nobody was kidding themselves; the Department of 
Defense was paying the bills for the ARPANET, universities for BITNET, and compa 
nies for CSNet. Nothing was free to the institution, but the users perceived the net 
works to have no incremental cost. This led to allocation by congestion, but I m not 
talking about economics yet. Regardless, a software author could give away software 
for the mere cost of uploading it to a distribution point. 

Freemacs was distributed from SIMTEL-20.ARPA, an FTP site with copies of most useful 
MS-DOS software. It was run by the Army at the White Sands Missile Base, but nobody 
cared about where it was physically. The point was that they had good stuff, and they 
were sharing it. This made me a contributing member of the open source community. 

Freemacs and Business 

Freemacs was a hobby, and I had no intention to turn it into a business. My comput 
ers (even my home computers) were paid for by my employer, so I had no expenses 
to cover. As the program gained users, they told other MS-DOS users about it. Those 
users wanted updates, and none were available on any of the worldwide networks of 
the day. Having no other recourse, they asked me for copies. I knew that they were 
gaining from these copies, so I asked for a portion of those benefits in the form of a 
copying fee. Between 1985 and 1991, most of the activity of Crynwr Software con 
sisted of putting software on floppy disks and mailing it to customers. 

Two interesting stories about mailing floppies: one is about a customer in Ireland who 
had two floppies go bad on him. Guessing that his email was going through some kind 
of antiterrorist scanner (as the Irish Republican Army was quite active at the time), I 
sent him a third floppy wrapped in Imm-thick lead foil. That floppy got through OK. 
Another is of a customer who, although a part of the defense department, had no Inter 
net access, or even have a modem and this was after almost everybody had gotten on 
the Internet, so I was surprised that they were even allowed to telephone out, but I sent 
them a floppy with software on it and they were happy with that. 

Freemacs and Business I * 139 



Staying in Touch 

It s crucially important to stay in touch with your users. The biggest advantage 
an open source developer has is close contact with users. If you re the primary 
user for the software, of course you know what users want you just sit and cog 
itate. Quite a bit of open source software is written to "scratch your own itch." 

This is easy and rewarding because very little communication is needed. Pro 
grammers are not typically great communicators (most programmers fall into one 
of the four NT classes on the standard Myers-Briggs personality test). A program 
mer who can listen and talk is worth her weight in chips (and chips are worth 
more per ounce than gold). 

With Freemacs, I started with a single mailing list, which proved to be a mistake: 
some people don t need any help and just want to know when new releases come 
out; other people want to get or give help but don t want to code; and still other 
people are interested in every miniscule detail of the program. One list cannot 
serve everyone. I found that I needed three lists. 

One list carried only announcements of new releases. You really want to have an 
announcements list, and you need to remember to use it. You might send only 
one or two pieces of email a year, but those are crucial. First, you need to remind 
people that they ve given permission for you to send them email. Second, people 
need to know that you re in business even if they don t currently need your busi 
ness. Any user might suddenly find himself needing to become a customer. You 
need to be the proprietor of the relationship between the software and the user, 
as you ll use that relationship to make money. 

A second list was for user-level help. Some programs are exceptionally powerful 
(I m not thinking of the Unix "cat" or "tail" here, but something more like send- 
mail or qmail) and in-depth knowledge to properly exploit all that power. Some 
users want to acquire that depth. Others do not, but will dip their toes into the 
depths by asking a question on the user mailing list. If there are sufficient users 
of the program, you will have other businesses competing with you. One of the 
ways they will compete is by offering to help other people. No need to worry, 
though! By virtue of your proprietary interest in the program, you will have a 
built-in advantage over these other businesses. In any case, customers like com 
petition because they perceive it as ensuring fair prices. 



140 * C Open Source and the Small Entrepreneur 



The third list is for developers. 1 am of two minds here. You could have the devel 
opers list open to all comers, regardless of their to contribute to the project. Or 
you could have the developers list be open only to those who actually have con 
tributed. The main tradeoff is protecting the time and attention of your contrib 
utors. You don t want them signing off the developer mailing list because it has 
too many user-level questions being asked on it. You really need their attention 
to help you make decisions that will affect them. For example, if you change an 
API, you need to clear that change with your developers first, because it keeps 
them involved in the process, and second, because they may be relying on some 
thing you re doing. 



FreeDOS has adopted Freemacs as its standard text editor, so it still has a user base. I 
only rarely do any MS-DOS work, and when I do, I m happy with the state of 
Freemacs, so it s now frozen in time. There were never any commercial users, so 
apart from selling copies on floppies, Freemacs managed only to buy me my own 
computer for home. 

Packet Drivers 

Crynwr Software came into its own with packet drivers. A packet driver allowed the 
sharing of an Ethernet card between two protocol stacks. For about a year, the only 
possible way to get Novell network clients on the Internet was by using a packet 
driver. Also, a packet driver would hide the differences among Ethernet cards. 

Unlike video boards, which are at least compatible at the VGA level, Ethernet boards 
have never been compatible. 

Back before packet drivers existed, there were network clients and servers largely 
Novell NetWare. The manufacturer- written network driver was linked to Novell s 
code in a single executable. The resultant program had no API for an external pro 
gram to send or receive an Ethernet packet, which was very bad for any competition 
to NetWare. Maybe Novell planned it that way, but I doubt the company was that 
Machiavellian. 

Anybody wanting to send packets other than NetWare s had a problem. 

The 3Com 3C501 was the market leader, but it was a very insufficient card. It had one 
buffer shared between transmit and receive, so a packet could be lost if it arrived when 
the buffer had not been emptied, cleaned, and turned around. However, everybody had 
drivers for it. Novell, in an attempt to improve the state of the art, took National s 8390 
demo board and put it into production as the NE1000 (and later as the NE2000). This 
board had sufficient memory with separate transmit and receive buffers. 



Just about then, other Ethernet controllers were coming on the market. People were 
using the Intel 82586, the AMD LANCE, and the National 8390 (in non-NE2000- 
compatible ways). Only NetWare included a device driver development solution. It 
had a driver development kit (DDK), and a certification house (Novell Labs). 

Other protocol stack vendors were doing the same thing producing drivers linked 
into their own products. No vendor had a driver that could be shared, however. 
While Novell and Microsoft pondered, little FTP Software (now owned by NetMan- 
age) had the same problem as everyone else: too much hardware and too few driv 
ers. It came up with its own specification for a shared Ethernet driver and, unlike 
other vendors, published it as an open standard. 

I was working for Clarkson University at the time, and we had the same problem as 
everyone else: how to support multiple pieces of software and hardware at the same 
time. I was using Phil Karn (KA9Q) s NOS, and he had packet driver support. So, I 
wrote packet drivers for the two Ethernet adapters in use at Clarkson (the 3c501 and 
Racal-Interlan NI5210) and published them as open source software. 

A number of fellow Internet users contributed drivers, and before long, we had cov 
ered a considerable portion of the industry. This led to more support from TCP/IP 
vendors, and a group at Brigham Young University wrote a NetWare driver that 
could use a packet driver. We really got the ball rolling then, because anyone with a 
NetWare network could put it on the Internet. 

There were some holdouts, notably Microsoft and Novell, both of whom started pro 
moting their own standards: NDIS and ODI, respectively. The NDIS document was 
published from the start, but there were no sample drivers, and no base of code from 
which to build. ODI documentation was available only with an expensive DDK pur 
chase. A packet driver distinguishes itself by coming with source code, by having a 
simple, approachable API, and by being small in size. The typical driver was 5K of 
executable, compared to 20K for ODI, and 40K for NDIS. 

By the mid- 1991, I realized there was money to be made providing packet driver 
programming and certification services the latter for drivers not written by Crynwr. 
So I left Clarkson and, after a five-month placeholder stint at a local PBX company, I 
started Crynwr Software. Up until 1998, Crynwr s main source of income was from 
packet drivers. 

Packet Driver Income 

Most likely, everybody has heard that the way to profit from open source software is 
to sell services. That s true, but there are many different types of services. I ll list 
some of them in the following paragraphs. 



142 * * Open Source and the Small Entrepreneur 



The first, and most profitable, is contract programming. Various people need packet 
drivers written, or features added, or bugs fixed. I contract with them to fix it, either 
for a fixed price (if I understand the problem), or at an hourly rate (if discovery is 
needed). Buyers don t like cost uncertainty they really like to know what some 
thing will cost up front but whenever you bid a fixed price, you are taking on the 
risk that the project will be much harder than you thought. 

I have actually been successful doing what appears, at first sight, to be the worst of 
both worlds: charging per hour with a minimum and maximum price. If you set the 
minimum and maximum to reasonably sane values, the risk is reasonably shared 
between the two parties. 



Business Tutorial 

Here s a quick tutorial, which I wish I had had when I started, on how people do 
business. First, customers expect to do business first, and pay you for it later. The 
customer accomplishes this by issuing a private currency called a purchase order 
(PO), with a face value and a serial number called the "PO number." Purchase 
orders owed to you are Accounts Receivable. Purchase orders you have issued 
and will pay are called Accounts Payable. I call a PO a currency because you can 
get a loan against good receivables, and you can sell bad receivables (customers 
who don t pay, or pay very late) at a discount. 

Never do work on a promise to pay you. If someone is really going to pay you, 
they ll be able to cut you a PO. People change jobs and companies go into bank 
ruptcy. If you have a PO number, that s as good as gold, because a company s 
ability to purchase things depends on its reputation for paying on terms. If it loses 
that ability, nobody will accept a PO from the company, and then it has to pay 
cash for everything. Get that PO! 

Some companies have intricate purchasing systems, where you have to be a qual 
ified vendor, you have to sign a W-9, and you have to sign a nondisclosure agree 
ment just to work with them. Other companies just whip out the credit card, and 
you re good to go. For any company larger than 50 people, though, you ll be 
dealing with a buyer. Most buyers are used to purchasing software as a product. 

Although they are starting to understand that software can be a service, you 
might still might run into a confused buyer, because sometimes they re told to 
"buy this software" and they don t understand that they re purchasing a CD and 
a support contract. Take the time to educate them about the difference, and you ll 
have an easier time working with them. 



Packet Driver Income X 143 



I ve also sold proprietary packet drivers, although this was a special circumstance 
(and one that was very profitable to me). I had a customer who wanted a new packet 
driver, but who didn t want to pay the entire price for it. He wanted to pay only half. 
He persuaded the vendor (SMC Semiconductors, now SMSC) to pay the other half, 
since the packet driver would be useful for all the vendor s customers. That seemed 
fair to me. I had his purchase order and SMSC s promise. Unfortunately, he paid up 
and SMSC didn t, so I had a packet driver that the vendor hadn t paid for. If I made it 
freely copyable, no vendor would ever bother to pay me, so I decided to license it to 
SMSC s customers until SMSC paid me. The company never paid, so I sold it with a 
clear conscience. 

I ve also dual-licensed packet drivers. A vendor that was going to embed an Ethernet 
chip into its product and use an embedded processor wanted to freely copy code 
from the packet driver without taking a chance that its driver would become a 
derived work under the provisions of the GPL. So the vendor purchased a copy of 
the code from me, licensed for any use except resale. 

I ve also sold compatibility certification. Digital Equipment Corporation had written 
its own proprietary packet drivers. DEC wanted me to certify that the company was 
compatible with the open source packet drivers. I had written a test program that 
would exercise the edges of a packet driver to try to break it. If that program ran, it 
meant DEC had made no stupid mistakes reading the specification. I also ran a stress 
test for several days; if that didn t run into problems, it meant DEC had made no stu 
pid coding mistakes. 

I ve also done pure consulting. Contracting is different from consulting. A contractor 
is someone who sells his work output, and a consultant is someone who sells his 
ideas. A customer wanted me to describe how my packet driver worked on their 
hardware, so they sent an engineer to my site for a day to get a debriefing. He took 
my family out to lunch, and I got paid handsomely for the day not as much as I 
would have been paid to write the improvements myself, but you can t make all the 
money all the time. 

Qmail 

Qmail 2 is an MTA written by Daniel J. Bernstein, a University of Chicago professor. 
He needed a fast MTA to run his mailing lists. While I was working on Freemacs and 
Packet Drivers, I was also running my own mailing lists. 



Qmail is not open source. It s freely copyable, but you can t redistribute modified executables. 
Although open source MTAs do exist, qmail is close enough to open source that I have 
insufficient reason to switch. 



144 x x Open Source and the Small Entrepreneur 



I experienced the same problems he did, so when he published qmail, I was on it 
right away. First, I ran it on a test machine, because I hate to lose email. Later, I ran it 
on my email hub after I learned to trust it. Slowly, the qmail community grew. 

Working with qmail has been a different experience from Freemacs and Packet Driv 
ers, as I did not write the qmail code. It s just as well, since that gives me something 
different to write about. It s a truism that you cannot sell something you don t own. 
A number of the techniques that I used for Packet Drivers do not work for qmail, as I 
don t own the copyright on it. Nonetheless, I have done quite well with qmail. 

Open source software serves to promote the author. By writing quality software 
(code and documentation), the author shows everyone the quality of his work. The 
software pushes his reputation out into the world. But what if you haven t written the 
software? I have found the best technique is to spend time reading user forums and 
answering questions. You can advance your reputation in this manner. 

There are other ways to establish your expertise with a particular program. You can 
contribute improvements to the code or documentation. You can write your own 
code that enhances the software, but that otherwise stands alone. You can also write 
extra documentation or maintain a web site about the software. 

I did all these things for qmail. Dr. Bernstein was not interested in registering the 
qmail.org domain, so I did it for him and maintained the web site, and I wrote a 
POP3 daemon for qmail and gave it back to him. I answered questions on the qmail 
mailing list. I wrote add-on packages and patches that people have found useful. 

Using all these techniques, I found paying qmail employment. Most often this came 
from people who needed qmail installed on their machines and had extra requirements 

that needed custom coding. Sometimes the customer wanted on-site qmail training 

which I have provided in Stockholm, Mumbai, London, New York City, Oslo, and 
Istanbul. All these trips were, of course, paid for by my customers, who also paid me. 

Open Source Economics 

It was my reputation, or "brand name" if you wish, that got me involved in the Open 
Source Initiative (OSI). I had been running the Free Software Business (FSB) mailing 
list since 1993, and had some success and reputation. When I heard about the Open 
Source name, I immediately adopted it in describing my software. It s so much easier to 
explain to customers why they should pay for software when it isn t "Free Software." 
Sometime after that, I heard that Eric Raymond was seeking to create an organization 
to promote OSI. I had been corresponding with Eric about open source, and we had 
discussed it on the FSB mailing list, so I volunteered to be on the board of OSI. 

I started the FSB mailing list so I could be more aware: I wanted to know what other 
people were doing to make money from open source. It seems that adding value to 



Open Source Economics * * W5 



things others created is a revolutionary way to make money; even Shakespeare might 
agree, as he routinely "recycled" plots and storylines written by other people. 

Certainly in the software business, an FSB is a new thing. With proprietary software, 
about the only way to add value is to sell the software for a higher price, or bundle 
products and sell them together for a lower price in toto. 

Running an FSB interested me in economics. When you give up a payroll check and 
start paying yourself, you also have to pay the business s share of Social Security 
taxes and start paying estimated income taxes quarterly. The government imposes 
this dead load on businesses. Why do they do this? Who benefits? Who loses? Eco 
nomics helps you answer these questions. 

Think carefully about how you price your services. There is an economic concept 
called "price differentiation." It means that you charge different parties different 
prices for slightly different services. If you charge a single price to everyone, and that 
price is too high, you miss out on helping some people who cannot afford your ser 
vices. If you charge a single price that is too low, you create value for some people 
beyond the amount they paid. It s not exactly fair that they should gain a lot more 
than you. To maximize your gain, sell things to people based on the value they 
receive rather than the cost to you. For example, everyone has seen software sold for 
less in a third-world country or to a school. 

Of course, pricing is also related to costs. Think about both transaction costs and so- 
called "sunk costs." Every transaction consumes a small amount of value in the transac 
tion itself. The buyer needs to evaluate whether the cost spent is worth the value 
received. The seller needs to take the money and provide the value. This cost is not 
received as value by either the buyer or the seller; it is simply wasted. One way you can 
sell support to a customer is on a self-renewing yearly support subscription. One 
month before the subscription expires, you send the customer an invoice. The sub 
scription portion avoids having to bill the customer for each support request, and the 
self-renewing portion avoids having to ask the customer to renew the support contract. 

A sunk cost is one you cannot recover by selling the thing you bought for example 
a railroad or a run of fiber optics cable. You can pull up the rails and sell them, but 
you won t get back anywhere near the cost of building the railroad. Likewise, spend 
ing money to create or improve open source software is a sunk cost. Once you ve 
spent that money, you have no way to sell the software (except by exiting the open 
source system by using a dual license). Just as a railroad needs to recover its sunk 
cost by selling transportation services, you need to recover your investment in the 
software by selling related services. 

One of the ways to manage costs is by making use of "public goods." A public good 
is nonrivalrous (meaning that my use doesn t affect your use) and nonexcludable (I 
can t stop you from using it). Absent copyright or patent protection, information is a 



146 ^ C Open Source and the Small Entrepreneur 



public good. Open source is typically copyrighted software, but is licensed under 
terms that make it effectively a public good. There is currently great debate over how 
much excludability is necessary to produce the optimal amount of software. 

Previously, economics students were taught that public goods were always underpro 
duced, with lighthouses as the canonical example. 

Someone dug up the history of lighthouses, only to find that the early ones were 
built by voluntary organizations. In a similar manner, the Free Software Foundation 
(FSF) wrote the GNU tool set as a public good. Economists can no longer assume 
that public goods are underproduced. 

People don t particularly care about products. People only buy things and own things 
for the services those things render to them. People don t want to own a washing 
machine; they just want clean clothes. Any desire to own a washing machine is second 
ary to having the clean clothes. The same thing goes for computers, only computers can 
provide many different services. The same services that someone can get from a soft 
ware product can also come from open source software and a support contract. 

Economists have discovered many principles helpful to the proprietor of an open 
source business. Nevermind the joke about 10 economists having 11 opinions. This 
chapter can but touch on the principles. (See the "For Further Reading" section at the 
end of this chapter for more information.) 

Where Do We Go from Here? 

I ve brought you up to my present life. Well, not quite. I received an inheritance 
from my mother, which, when invested in the stock market, generates sufficient 
income that I no longer need to work for a living. I still take interesting jobs as they 
come up, and I support long-term customers because they ve become friends, not 
just customers. In everything else, I just write open source and distribute it as I wish. 
My advice to you is to always pay your retirement fund first: put 10% of every 
project s check into a brokerage account and invest it in an Exchange Traded Fund. 

Happy hacking! 

For Further Reading 

For a basic (and enjoyable) introduction to price theory, read Hidden Order, by David 
D. Friedman (Collins, 1997). In fact, read anything written by any member of the 
Friedman dynasty: Milton, Rose, David, or Patri. 

For a basic introduction to economics, read Economics in One Lesson, by Henry Hazlitt 
(Three Rivers Press, 1998). 



Where Do We Go from Here? *C M7 



For an explanation of the proper function of the law, read The Law, by Frederic Bas- 
tiat (Foundation for Economic Education, 1987). It s in the public domain, so you 
can find it on the Web, in print, and as a free audio book. 

For a very deep exposition on economics, read Human Action, by Ludwig von Mises 
(Fox & Wilkes, 1996). It s available at http://www.mises.org/humanaction.asp. 

The FSB mailing list is at http://crynwr.com/fsb.html. 
Freemacs is at http://www.freedos.org/jhall/freemacs. 
Packet Drivers are at http://crynwr.com. 
qmail is at http://qmail.org. 



148 x x Open Source and the Small Entrepreneur 







CHAPTER 10 
Wendy Seltzer 



Why Open Source Needs 
Copyright Politics 



Some programmers and businesspeople draw a distinction between "Free Software" 
and "open source." Free Software is political, they say, and open source is pragmatic. 
Free Software developers want to recede the world; open sourcers just want to write 
good code. This distinction is, of course, exaggerated. Many people adopt these 
labels for their own reasons; some switch between them depending on audience or 
context. But even the most apolitical of open source developers and users should be 
concerned by the copyright battles waged right now. The copyright law being made 
and enforced today will impact the software we can develop and use for decades, and 
its impact reaches far beyond commercial media. 

Imagine, for example, that you d like to build an open source home multimedia 
server. Nothing fancy yet, just a place to play music, watch the occasional DVD, and 
record television programs one machine to replace the menagerie of devices nest 
ing in your media center. Easier developed than cleared legally. Technically, you (or 
others willing to share with you) will be able to meet the challenges with Moore s 
Law-fast processors, ever-cheaper massive storage capacities, and clever user inter 
faces. The legal obstacles are harder to hack. Start with the music. If you have stan 
dard CDs, you re all set: plenty of Free programs let you play them from the CD 
drive, rip them to Ogg, FLAG, or MP3 (with a nod, perhaps, to the patent licensors at 
Fraunhoffer). Try to connect to a streaming service or purchase music online, and 
things get tricky. Apple s iTunes, the "new Napster," and Rhapsody all lack open 
source clients, and none would be happy with reverse engineering to write one that 
plays the music they sell encrypted or by subscription. 



Yet music is the easy part. Want to write a player for DVDs you ve purchased or Net- 
flixed? Because only closed source implementations have been licensed to decrypt 
the DVD s files, any DVD player you write is liable to be deemed a "circumvention" 
by the movie studios and courts, even if the only features you write match those of 
WinDVD or the standalone player under the TV. Television, then; recording over- 
the-air broadcasts shouldn t be too hard, since those are unencrypted. Watch out for 
the digital television transition and the broadcast flag, though. Unless public interest 
groups challenge succeeds, the FCC s broadcast flag rule will ban open digital TV 
tuners that can be used with open source software. The only ones who will be able to 
play will be those making closed hardware or proprietary software decoders. You ll 
encounter these obstacles before you even try to take any of your media off the server 
to exchange with friends or family. 

OK, but say for a moment that you have no interest in multimedia. Leave that mine 
field for another day and move on to business networks or productivity software. 
Even there, copyright law intrudes. Your security analysis of a system s encryption 
might be limited by what media companies have preemptively claimed as "techno 
logical protection measures;" your selection of replacement parts or add-on modules 
could be dictated by copyright-based tying more than fitness for use; your ability to 
interoperate depends on whether reasonable interpretations of the law prevail over 
some vendors extreme copyright claims. 

Like a group of once-healthy cells grown out of control, copyright law has metasta- 
sized to threaten the system of creativity it was once helping to support. No longer a 
"limited monopoly" to encourage creativity and dissemination of creative works to 
the public, copyright has become a blunt tool of exclusion, chilling development of 
software, among other creative endeavors. And so the fight to restore balance to 
copyright law cannot be dismissed as mere politics. Unless users and developers of 
open source software join the copyfight, they will find the new reality of copyright 
law restricting not just their freedom to play blockbuster movies, but also their core 
freedoms to write and run independent and interoperable software. 

From Movable Type to MovableType 

A balanced copyright law is enshrined in the U.S. Constitution: "to promote the 
progress of science and useful arts," Congress was empowered to grant authors 
exclusive rights "for limited times." The monopoly created was limited in time and 
scope. The first copyright law gave authors a 14-year term, renewable once, to pub 
lish and vend maps, charts, and books. Copyright protected original expression for a 
short time, while leaving others free to build around that expression (translations and 
dramatizations, for example), and then to recycle works entirely from the public 
domain once copyright expired. 



ISO ^ C Why Open Source Needs Copyright Politics 



Copyright law has changed with the introduction of new technologies. New means of 
reproduction often first challenge the copyright framework, then establish them 
selves as new creative tools for authors and their public audiences alike. At the turn 
of the last century, printers bought single copies of sheet music and punched holes 
into rolled paper to program "piano rolls" for then-new player pianos. Composers 
and music publishers sued, seeking to rein in this appropriation. When the courts 
held that punched paper didn t "copy" inked notes, Congress updated the law with a 
compromise not to ban player pianos or mechanical reproduction, but to permit 
anyone to produce piano rolls if they paid a "mechanical license" royalty for every 
roll sold. As the player piano market grew, more music reached more people, and 
more composers got paid for creating it. 

The pattern has repeated itself many times since. Songwriters and performers 
denounced radio until both found that it could promote sales. Movie studios deplored 
the videocassette recorder, saying Sony s Betamax would be the "Boston Strangler" to 
their industry. When they failed to shut Sony down, however, the industry converted 
its peril into a profit center, finding that viewers with home recorders were potential 
customers for rentals and sales of appropriately priced videotapes. Meanwhile, the 
Supreme Court s ruling that technology makers would not be liable for users copy 
right infringements so long as their devices were "capable of substantial noninfringing 
use" fueled a technology boom. The public and the creators shared the benefits of new 
technology the public could record movies from television to videotape; studios 
could sell or rent videocassettes more easily than reel-to-reel. 

Despite making it through these earlier transitions, the entertainment companies 
haven t stopped fighting technological change and the competitive threats it repre 
sents. The MP3 player is a slightly more convenient cassette deck, and the weblog is 
just the next step forward from the typewriter and mimeograph. This time, however, 
the entertainment industry has swayed many in Congress and the courts to the view 
that "digital is different," and induced them to change the law in ways that are differ 
ent and dangerous. 

This expansion of copyright s control interferes with open source development. The 
changes manifest themselves in layers, most notably overassertion of protection for 
code itself; excessive protection of other copyrighted content that code is dealing with; 
and misuse of copyright to control markets and maintain cartels in technologies of dis 
tribution or manipulation of code and content. Together, the copyright layers build a 
shell around not only proprietary code, but also around culture and innovation. 

Copyright in Code 

Copyright protects original expression in code, as it protects any other "original 
works of authorship fixed in any tangible medium of expression." Some developers 
use that protection to enforce the openness of their code, as with the GNU General 



Copyright in Code X 151 



Public License (GPL); others use it to reinforce a proprietary distribution model. But 
while copyright protects code s creative expression, it does not protect the functions, 
methods, or procedures that expression implements. Thus, even for closed code, a 
programmer remains free to study interfaces and functionality, free to interoperate or 
replace that code with his own. 

Situated as it is in an environment filled with proprietary software and poorly docu 
mented interfaces, open source development frequently relies on reverse engineering 
to fit in. Anyone who has used Samba to bridge Windows and non- Windows net 
works has benefited from Andrew Tridgell s reverse engineering of the Windows pro 
tocols for network services; anyone who exchanges files to read and write them in 
OpenOffice.org rather than Microsoft Word appreciates the reverse-engineering- 
derived ability to edit files in Microsoft Word format. 

Courts have long held that reverse engineering, the practice of examining something 
and taking it apart to figure out how it works, is a fair use, not an infringement of 
copyright. Even when programs are released only as closed, binary code, program 
mers can often discover a great deal about them by watching their operation or the 
file formats they use, or by disassembly. Reimplementing those discoveries in new 
code comports with copyright too. So, companies have been protected in taking 
apart a video game console to build a console emulator (Sony v. Connectix) or disas 
sembling a game to build a new one compatible with the console (Sega v. Accolade). If 
someone builds a better mousetrap after examining those that exist, copyright law 
will not stop her from deploying it. 

At least that s the black-letter law. In practice, though, many copyright holders try to 
extend copyright s limited monopoly to block reverse engineering, through a combi 
nation of copyright, contract, and anticircumvention. For example, Blizzard, maker 
of the popular Starcraft and Warcraft video games, claims that all players have 
"agreed," through click-wrap licenses they encounter before the programs run, to a 
contract that prohibits reverse engineering the Blizzard games. This contract plus 
copyright, Blizzard asserts, prevents anyone from interoperating without permission. 

The bnetd project began when a group of programmers became frustrated by the 
poor performance of Blizzard s battle.net server for multiuser play of the games they 
had bought. Instead of putting up with the frequent downtime and rampant cheat 
ing on Blizzard s server, they started work on their own open source game server. By 
watching the communications between game and server, the bnetd programmers 
were able to reverse engineer their own compatible server, which they set up to give 
owners of Blizzard games an alternate place to meet, and made the source available 
for others to join the effort. The public got another option for playing Starcraft, and 
another reason to buy Blizzard games. 

Blizzard rewarded the bnetd team s creativity with a lawsuit claiming, among other 
things, copyright infringement and breach of the click-wrap contract. The team had 

152 * C Why Open Source Needs Copyright Politics 



not seen any of Blizzard s source code, much less copied it, as they reimplemented 
uncopyrightable functionality, but that didn t stop Blizzard from pulling out the 
copyright sword. Copyright s lack of protection for functionality is deliberate and 
sound innovation policy the public benefits from being able to choose among com 
peting implementations of functionality, be they game servers or network services 
yet Blizzard follows in the steps of many trying to get around copyright s limits with 
contract claims. 

Blizzard was trying to limit the code others could produce, to extend the copyright 
protection on its own code. But many of those pulling out copyright s swords aren t 
trying to protect code, but other copyrighted content touched by or that they re 
afraid will be touched by code. These efforts, assertions of secondary liability, anti- 
circumvention regimes, and attempts to impose technology mandates, all limit open 
source developers ability to produce new code, to learn from old code, and to com 
pete in the market with proprietary code. 

Secondary Liability 

Copyright law has not been strictly limited to the direct infringers, but may be 
extended to those who "contribute" to the infringement in an ongoing relationship 
with the infringer, or with special-purpose equipment suited only for infringement. 
Thus, the proprietor of a hall, who looks the other way while paying guests listen to 
unlicensed music, can be held vicariously liable for the infringing public perfor 
mance, and the seller of tapes of a length precisely timed for copying particular copy 
righted albums could be held contributorily liable for the subsequent infringing use. 
Extended too far, however, secondary liability chokes off innovation. 

Twenty years ago, the Supreme Court rejected Universal and Disney s "unprece 
dented attempt to impose copyright liability upon the distributors of copying equip 
ment" with the ruling that manufacturers of devices "capable of substantial non- 
infringing use" could not be held secondarily liable. The MGM v. Grokstcr lawsuit, an 
attack by all the major record labels and movie studios against Grokster and Stream- 
cast, maker of the Morpheus filesharing software, is nothing short of an all-out 
assault on the Sony standard. 

The studios argue that Grokster and Streamcast should be liable because many users 
of the peer-to-peer software infringe copyrights notwithstanding that many others 
transfer public domain works from Project Gutenberg or the Internet Archive; freely 
licensed works including open source software and Creative Commons-licensed 
media; or government works. They argue that the producers of software should be 
held liable for its "predominant" or "principal" use. Their standard is unworkable 
both to an entrepreneur financing an untested product and to an open source devel 
oper releasing software, any of whose users could adapt it to an unintended, infring 
ing purpose. 



Secondary Liability *C 153 



Under the Betamax standard, makers of multiuse devices such as the VCR could thus 
offer them to the public without fearing that they might be held liable if customers 
misused them to infringe copyrights. With this assurance, hardware makers built 
components with open interfaces, including CD and DVD burners and massive hard 
drives, without fearing that someone might put the Plextor on the copyright hook by 
using that CD burner for large-scale copyright infringement. They built copying 
devices to transfer content. Software makers, too, have safely offered highly config 
urable and open source software with relative confidence that their users configura 
tions won t land them in hot water. The studios proposed redefinition threatens that 
freedom to innovate. 

Grokster is thus much bigger than peer-to-peer. An expansion of copyright liability, 
with a "predominant use" test, would make it safer to produce limited-purpose, non- 
user-modifiable devices and software than open hardware and open source software, 
regardless of the intent of the developer. 

If secondary liability for those who "contribute" to infringement in some ill-defined 
way weren t enough, the entertainment industry is likely to return to Congress with 
pleas of renewed urgency to pass the INDUCE Act, which stalled last term. That pro 
posed bill would add yet another level of indirect liability: "inducing" infringement of 
copyright would extend beyond those who made the tools, to those who explained 
how to make them work. Watch out that your documentation isn t too thorough! 

Anticircumvention 

The next stage of copyright expansion beyond direct infringement is the anticircum- 
vention and antitools provisions of the Digital Millenium Copyright Act (DMCA). In 
the real world, these provisions do little to stop hard-core piracy, but they present a 
serious barrier to open source compatibility. 

Section 1201 of the DMCA prohibits "circumvention" of technological protection 
measures controlling access to copyrighted works, and it bans manufacture, distribu 
tion, and trafficking in devices, products, components, or services that are promoted 
for, primarily useful for, or designed for circumvention of technological protection 
measures. Now, a copyright holder who employs a technological lock, such as sim 
ple encryption of content, gains the ability to control who and what programs or 
devices can unlock it, as part of the new right to control "access" to a protected work. 

This is an important functional change from the world of printed books or even CDs, 
where anyone who purchased or borrowed the physical item had the right to use it 
as she chose read the last chapter first or play the CD in the car or the computer. 
Someone who wants to develop a new shuffle mode for CD playback, or a new rip 
per with better compression for transfer to a portable music player, can do so with 
out seeking permission from the recording companies. It s the misuse of those 



154 * x Why Open Source Needs Copyright Politics 



tools say, to copy CDs and sell the copies without authorization, that can be pur 
sued as an infringement of traditional copyright. 

Under the new anticircumvention regime, however, those who control copyrights 
can take their control much further. Thus, copyright holders say, and courts have 
agreed, anyone who develops a decoder for an encrypted format without a license is 
producing circumvention tools in violation of the DMCA. So, groups of copyright 
holders, who couldn t individually control markets, join together in licensing cartels 
backed by the magic of the DMCA, by which they control not only copyrights but 
also the surrounding player technologies. 

And so it is with digital video discs. Movie studios and consumer electronics and 
technology companies developed the content scrambling system, a.k.a. CSS, applied 
it to DVDs, and declared it to be a technological protection measure. By forming the 
DVD Copy Control Association to license the CSS specification as a trade secret on 
restrictive terms, they sealed themselves a nice cartel, simultaneously protecting 
themselves against disruptive innovation in video players from outside of the estab 
lishment and walling off their copyrighted works against fair use copying. 

The CSS encryption was trivially easy to break, once Jon Johansen and some Ger 
man programmers set their minds to it. But that s beside the point, since the law pro 
tects even weak technological protection measures with the full panoply of anticir 
cumvention and antidevice prohibitions. Even weak measures set up the law s sharp 
division between licensed access and unlicensed circumvention. Thus, with DeCSS 
and its successor code, anyone can play or rip a DVD on any platform, but with 
DMCA, no one can lawfully do so in the United States, or even develop code for 
DVD playback, without a license from the DVD-CCA. 

Kaleidescape is a small company with a rich clientele owners of hundreds of DVDs 
willing to pay nearly $30,000 for a DVD jukebox to organize and store them all. 
Kaleidescape built this machine a computer filled with massive hard drives onto 
which customers could rip their DVD movie collections. For its efforts to help the 
movie industry s best customers get more out of their purchases, Kaleidescape earned 
itself a lawsuit from the DVD-CCA, claiming Kaleidescape had breached its contract 
licensing the not-so-secret DVD trade secrets. 

Kaleidescape s system, which allows the creation of persistent digital copies of the 
content of DVDs and allows copying of the CSS copy protection data, is not designed 
in a manner and does not include features clearly designed to effectively frustrate 
efforts to defeat the copy protection functions.... For these reasons, Kaleidescape has 
breached the CSS license. 

Without that license, even if similar information could be derived from reverse engi 
neering and publicly available information, using it to enable DVD "access" and play 
back would be labeled circumvention. 



Anticircumvention X 155 



As the movie studios have done with DVDs, record labels and software companies 
have done with multiple incompatible formats for streaming and downloadable 
music: Windows Media, Janus, Apple s FairPlay. Many of these have succumbed to 
reverse engineering of varying degrees of sophistication, from "burn it to CD and 
rerip," to "run it through a simulated sound-card driver," through cryptanalysis 
(much of it from Jon Johansen, again). 

Yet, like CSS, none of the music protection measures is licensed on terms that per 
mit open source development, nor could they be. Open source is incompatible with 
both their stated aim, to prevent "piracy" of content, and the unstated underlying 
goal of technology control. Since the essence of open source is user modification, 
users could easily modify in or out any particular features and the first to go would 
likely be the restrictions of digital rights management (DRM) and barriers to interop 
erability. Even if open source version 1.0 incorporated all the restrictions of propri 
etary clients, numerous versions 1.0.1 would likely disable them. Thus, anticircum- 
vention regimes lock open source out of mainstream development of entire classes of 
applications to interact with these copyright cartels media. 

Of course, it s not just open source developers who need reverse engineering. Con 
sider RealNetworks attempt to sell music that would play on the popular iPod. Real 
could have transcoded downloads into standard MP3, which play on the iPod, but 
Real (or its record-label-relations department) wanted to include DRM on the files. 
So, it reverse engineered Apple s FairPlay format, in a move Real called Harmony, to 
encode Real files for the iPod. Apple fought back with legal and technical threats. 
Along with intimations that it might use the DMCA, Apple issued a statement that 
said, "We strongly caution Real and their customers that when we update our iPod 
software from time to time it is highly likely that Real s Harmony technology will 
cease to work with current and future iPods." Real wasn t threatening to infringe 
copyrights, but to give customers a way to interchange their iPods for other devices. 
Once again, DRM is market protection, not copy protection. 

The Threat to Research 

Again in the DMCA, copyright bleeds beyond entertainment media and content. 
Here, it also chills research into encryption that may be used to secure copyrighted 
works, regardless of whether that research touches entertainment content directly. 

When the Secure Digital Music Initiative (SDMI) invited programmers to "attack" 
security technologies they were promoting to control digital music, Princeton com 
puter science professor Ed Felten and his team stepped up to the challenge. They 
broke the security and prepared a scientific paper analyzing the weaknesses of digi 
tal audio watermarking. By scrutinizing these implementations, they could help the 
public, and particularly those considering watermarks to protect their own materi 
als, to evaluate the security of watermark technologies. 



156 * C Why Open Source Needs Copyright Politics 



But when the Felten paper was accepted for presentation at a computer security con 
ference, its authors were threatened with a lawsuit from the Recording Industry Asso 
ciation of America (RIAA) and SDMI s technology providers. RIAA and the technol 
ogy companies claimed that even the scientific analysis of flaws in security 
technology "would subject [the] research team to enforcement actions under the 
DMCA and possibly other federal laws." The RIAA suggested that the paper and con 
ference presentation fell under the DMCA s antidevice prohibition, which bans offer 
ing to the public "any technology, product, service, device, component, or part 
thereof," designed or marketed for circumvention or having only limited commer 
cially significant other purposes. Facing legal pressure on conference organizers and 
researchers, the team withdrew the paper. 

While Professor Felten and his team ultimately published a version of the paper, 
"Reading Between the Lines: Lessons from the SDMI Challenge," they were forced to 
strip out technical detail. With the vague anticircumvention threat hanging, the 
authors felt they had to omit code samples that could be construed as aiding circum 
vention even though that code would have helped other researchers and develop 
ers to understand the watermarks weaknesses better and to learn from SDMI s errors 
in building their own security systems. 

The work of the Felten team never infringed copyright. The researchers did not copy 
a single piece of music without authorization, nor even produce tools that enabled 
others to do so directly. And yet, they were caught up by the DMCA s vagaries 
because their paper intended to educate other researchers and developers of secu 
rity systems might also have provided "part" of a tool for circumventing a copy 
right control. Will the same hurdles rise before someone who builds a tool to strip 
accidental copy protection from fonts he himself has created; security researchers 
who find vulnerabilities in network software they analyze; someone describing hard 
ware modifications to make the Xbox a more general-purpose device? So far, the 
answer has been "yes" for Tom Murphy, SNOsoft, and Bunnie Hwang. 

For those who are attracted to open source development because of its opportunities 
to learn from and share with others, this antiresearch aspect of the DMCA should be 
particularly troubling: research is being chilled precisely because it teaches too much, 
because its teaching might be misused. Of course, closing systems doesn t stop them 
from having security flaws, or even prevent those flaws from being discovered, and it 
does block some of the most effective information sharing around better security. 
Both the teaching that open source developers depend on to improve their programs 
and the teaching of open source code itself are at risk. 

Further, the DMCA has been used in attempts to block competitive interoperability 
of devices including printer toner cartridges and remote control garage door open 
ers, as manufacturers add little scraps of code to devices and hope to leverage its 



The Threat to Research *> 157 



"protection" into market control. Though those arguments have been rejected so far, 
it s unlikely we ve seen the last of them. 

Technology Mandates 

The final layer of copyright s expansion, so far, is to technology mandates, where an 
entire technology is redesigned by government fiat in the name of copyright protec 
tion. Senator Ernest "Fritz" Rollings proposed one of these in 2002 that would have 
required every "digital media device," including the personal computer, to be rede 
signed to protect copyrighted content. While the "Fritz chip" never came to be, a 
smaller version has been foisted upon us in the form of the Broadcast Flag, an FCC 
rule set to take effect July 1, 2005. 

With the government eager to get broadcasters off the valuable analog spectrum and 
onto digital transmissions, movies studios threatened that they wouldn t allow their 
content to be broadcast digitally in the clear, and warned that there would be no transi 
tion without their "high-value" content. The FCC didn t want to abandon the notion of 
unencrypted over-the-air television broadcasts, but it did want to give the studios their 
"protection," so it put the restrictions into the hardware. At the studios recommenda 
tion, the FCC adopted a rule that adds a "flag" to these unencrypted broadcasts and 
then requires every receiver to watch for the flag and output flagged content only to 
"compliant" devices or in low resolution. Only devices that can implement DRM in a 
manner "robust against user modification"