|
|
Towards Organic SoftwareCees de Groot$Revision: 1.2 $
This paper discusses why software engineering is a bad idea.
It talks about
some interesting developments in science and how they should affect
our view towards software development.
Copyright (C)1999, Cees de Groot Reproduction of this publication without prior permission
by the author prohibited. Since 1986, people have been paying me to write
software. Since 1986, I've mostly made a mess of it, and the only reason
people still are paying me to write software is the fact that my mess
is at worst the same, but luckily often less a mess than the mess my
colleagues make. That's what makes you a good programmer: not fouling
up too much. This is an exaggeration, of course. However, this industry's
state is a sad one: software is just as buggy as it used to be, and
users haven't become measurably more productive. At the same time,
businesses are throwing awful lots of money at solving the same
problem: making information available and usable. [Brooks95] already told us that there is no silver
bullet. There is no magical method of making things work like they
seem to work in "other engineering disciplines". I quote that because
it implies that making software is an engineering discipline, and I
don't think it is. At least, not one that is comparable with the
standard engineering disciplines, which seem to have gotten their act
reasonably well together (when is the last time your car
crashed?). This paper attempts to analyze why progress has been so slow, and
tries to reason from first principles why current software engineering
methods don't work very well.
Most of the time since Galileo and Newton, science has
been working under the assumption that things behave in an orderly,
predictable manner. If you want to know where and at what time Venus will rise
tomorrow, you can just plug some values into some equations, press on
the "Calculate" button, and read the answer. This
attitude started with Newton discovering the mathematical formulae
behind gravitation and other basic mechanics, and it has spread out to
most other sciences as well. The assumption is that nature follows
laws, you can deduce these laws from observing behavior, and then you
can make predictions about what will happen by plugging values into
these laws and calculating the results. Modern science has achieved
most of what we know today by following this framework, so at the very
least it seems a workable model of how nature operates. One of the
prime reasons that governments and businesses pour enormous amounts
of money in scientific research is indeed the very hope that models
will result, allowing the Maecenas to use the model to make
predictions - about whether the moon lander has enough fuel to get
back, or whether raising interest will help to temper inflation, whatever.
However, all these models assume that systems behave in a
linear manner. They assume that if you plug in more-or-less the
correct initial value (say, for the gravitational pull of the moon and
the weight of the moon lander), you get more-or-less the correct result
(the amount of fuel needed). Add a small safety margin, and you're all
set. Edward Lorenz, a meteorologist, discovered in 1960
that this needn't be true. What he was working on was a simple
model of the world's weather. He left out most of the details like
snow and rain, and concentrated on the simplest of simplest: movement
of air in the atmosphere. One day, he wanted to restart a simulation
but as he didn't want to wait too long, he just punched in the numbers
the computer printed out as the initial conditions for the next
run. Coming back after a cup of coffee, he thought he'd made a mistake
entering the numbers because the weather model was completely
different from the first run. But no, the numbers were correct. Then,
he realized that, effectively, he had been rounding off the numbers:
the computer printed three decimals, but internally worked with six
decimals, which meant that the second model differed not more than one
part in a thousand from the original model. It seemed that his simple
set of equations was so sensitive to initial conditions, that the
model took off in a completely different direction. This was the end of a meteorology dream (long-term weather
prediction), and the beginning of a new science. It was one of these
things for which the time was right, because numerous people from all
kinds of disciplines came up with the same results: non-linear
systems, often deceptively simple, could show unpredictable,
chaotic behavior. Something as simple as
iterating over a quadratic function and looking where the result would
go resulted in something as complex as the well-known Mandelbrot
fractal. Suddenly, people realized again that the linear models
used in standard scientific practice were just that: models. When
calculating where Venus would rise, they were using a model to make a
prediction—a very accurate prediction, but still a
prediction. Furthermore, it dawned upon scientists that a lot of
"noise" that they would tend to filter out of their experiments were
actually the results of non-linear, chaotic behavior. The 1980's were
the time that top-notch physicists forgot about expensive particle
accelerators and started studying basic, "high school student" things
like the behavior of a dripping water faucet.
The Santa Fe
Institute is probably the center of the world for a new
discipline, called Complexity
Science[2]. Founded
in 1986, it is meant to be a basic research institute that studies
complexity, the stuff that has made life, societies, and other complex
organizations possible. The basis for complexity is a reversal of
the classical science view, which is based on dissecting stuff until the
basic elements are found (at the forefront, of course, is particle
physics, but biology, economy, etcetera all basically do the same
thing). Complexity arises in the other direction: when basic elements
are added together and allowed to interact in certain ways, their
interactions tends to form something that is discernible at a higher
level as a pattern that cannot be explained from the elements. It's
the old reductionism versus holism all over again, with the
not-so-small difference that computer simulations, a relatively young
entry in the scientist's tool chest, seem to prove the validity of
this particular form of holism over and over again. Let me give two well-known examples to make this point
clear. First, John Conway's game of Life. It's not really a game,
technically it belongs to the class of simulations called
Cellular Automata, and the rules are very
simple:
If a cell has 2 or 3 filled neighbors (out of the
nine possible neighbours), it will be
filled in the next round; Otherwise, a cell will be empty in the next round.
Put up a checkerboard, put some stones on it ("filled cells") at
random, and start playing. Better, get one of the zillion computer
simulations - almost every programmer on Earth has implemented The
Game Of Life one time or
the other (the editor I'm writing this in has a built-in version!).
These simple rules for individual cells can lead to amazing
patterns of groups of cells (the higher level), and even some 25 years
after Conway came up with it people are still amazed by this
game. Whole catalogs of patterns of cells that do interesting things
exist. While watching a Life simulation, you cannot help feeling that
something on that screen, well, lives. In 1986, Craig Reynolds made a computer model for the kind
of movement behavior a flock of birds or a school of fish shows:
intricate patterns that seem to steer a bunch of organisms as if it
were a single organism avoiding obstacles or just bouncing around. As
for Life, Reynolds found that only a few simple rules sufficed to model this
behavior:
Separation: steer to avoid crowding local flock mates; Alignment: steer towards the average heading of local flock mates; Cohesion: steer to move toward the average
position of local flock mates.
The results of these three rules can be seen in every modern animation
movie where flocking animals feature (be it bats in "The Hunchback of the Notre
Dame" or a herd of gnus in "The Lion King"). Again, a set of simple basic
rules about how elements could interact was enough to cause intricate
patterns on a higher level. Over and over again, this point has been shown. We're living
examples: a reasonably simple set of rules (interactions between
molecules—chemistry) was enough to trigger a sort of upwards
avalanche called evolution, where chemical cross-breeding between
molecules (now and then helped by mutations through cosmic radiation)
and natural selection created the most complex things we know of, like
human beings and whales. Emergent, bottom-up complexity seems to be how nature puts
things together. Even though on a scientific level a lot of the work
is preliminary and unproven, the pattern is too clear to
ignore.
How do order, chaos and complexity fit together? Chris
Langton, the father of Artificial Life, has
come up with the idea that Complexity sits on a thin line between
Order and Chaos, like the molecules between solid and fluid in a phase
transition. In fact, he has shown a lot of analogies, both qualitative
and quantitative, between some physics thingy called "second-order
phase transitions" and the division of systems behavior into Order,
Complexity, and Chaos. By now, a lot of people are convinced that there is a
driving force of nature towards complexity, just like there is a
driving force towards nothing at all (worded in the Second Law of
Thermodynamics). There is no rigorous proof yet, but the idea is
simple: say that there's a completely orderly system of elements
interacting in some way, with the possibility for the elements to
adapt their behavior to one another. Now, if some
external or internal influence gives a little kick to one or more elements in the
system, it might give this element an edge over the others and this
behavior might spread. In an economy, a company may come up with an
invention like, say, the transistor or the integrated circuit. This
invention gives rise to new industries, people will leave companies
and setup their own, and on the whole will result in a ripple effect
through the whole economy, bringing the system to a less orderly state overall. Now, imagine a system that is in a
chaotic state. Here it is even clearer to see that if elements of the system
start to interact in patterns, they have an edge over the rest. The
fact that people organizing and specializing had an edge over the
people still roaming about collecting their own food has led to the
rise of societies. So, orderly systems tend to get less orderly, and
chaotic systems tend to get less chaotic. Right in the middle, there's
a stable situation of complex systems. Adapting systems to internal
and external influences so that they become complex systems seems to
be nature's way of dealing with organization.
A holy grail of software engineering seems to be to mimic
other engineering disciplines, like electronics or building. Although
there are lessons to be learned from these disciplines, I maintain
that it is impossible to find an analogy that holds on closer
inspection, and that it is therefore not wise to base software
engineering discipline on analogies from other engineering
disciplines. There are a couple of aspects that are not present (or not on the
forefront) in "hardware engineering" that are present in software
engineering: testability, interaction complexity and malleability. I'll
detail them in the next few sections.
When the mayor opens a bridge, one thing he doesn't do a
week later is to go to the bridge engineers and ask them "You know, this
bridge is basically OK but could you please give it round pillars
instead of square pillars? I think it looks better." No, a bridge is
planned in advance, built, and unless there is something seriously
wrong with it the bridge is a take-it-or-leave-it deal. That's why
engineers take a couple of years to design the bridge, make scale
models, do wind tunnel tests, etcetera - whatever it takes to validate
the design in all its aspects is done because building the bridge is a
one-shot action: it must succeed, because there's no way to beta test
it. Software, on the other hand, is pure brain stuff. The
hardware is completely unimportant, and things like the language the
software is expressed in are unimportant as well. Software is thoughts
and concepts in some convenient notation, and this notation typically
lives in formats that make it easy to modify (like in the format of
text files on hard disks). Software is almost infinitely malleable. The
first thing a user does when starting to work with new software,
especially with new custom software, is think about what needs to be
changed and often pushing the supplier to make these changes
("yesterday would be fine, thank you"). The good thing here is that software builders get a second
chance. If things don't work out, they can fix it. The bad thing, of
course, is that they get a second chance—endless design,
prototype and test cycles aren't seen as necessary, because if the
user has a problem, it can be fixed. In fact, the pace of the whole
information technology industry is such that pre-production activities
taking years to complete, like in bridge building, are simply
unacceptable.
How do you test whether a car has certain aerodynamic
qualities? Simple: you take a lump of clay, form it according to the
design drawings, put it into a wind tunnel, and let sensors and
computers to the calculations. The end result is a coefficient - if
it's too high, you try to make little changes to get it down, and if
it's way too high, you send the thing back to the drawing
board. How do you test software? Alan Turing came with a proof,
more than half a century ago, that you cannot deduce whether a
non-trivial program
will finish its task or not without actually running the program and
sitting it out.[3] Later work has shown that it is
impossible to perform complete tests on software that reacts on users or
networks. Software is just too complex, there are too many states and
inputs, and the whole thing exhibits highly nonlinear behavior: you
can check the behavior for value X, but that doesn't guarantee you
that the behavior for (X plus a tiny bit) will be the same. Furthermore, because of all this, there's no way to make a
scale model of a piece of software and put it in the wind tunnel. Even
with electronics engineering, you can make a model and exercise and probe it
before actually starting up the chip factory, but with software there
is no such thing as a model you can test - the model is the software,
and vice versa. Some schools of thought in software engineering maintain
that you can test software, by testing the components. The reasoning
is based on reductionism: you take the problem apart, build low-level
components, and solve the problem by bringing the components
together. Testing is done on the low-level components and, they say,
this says something about the high-level software because in the
reductionist thinking, the whole cannot be more than the
parts. Software models like Life and Boids prove otherwise, I
think.[4]
A last difference is the level of interaction between
components. Because software runs mostly on digital electronic
computers (out of convenience, not out of necessity), software
engineers see in electronics engineers their close cousins. Moreover,
electronics engineering has the nice property that a lot of the work
consists of plugging existing components together, not in constantly
reinventing and redeveloping the components themselves. Lots of the
work in electronics engineering consists of building higher-level
components out of low-level components, and using these to build even
higher level components, etcetera. It seems natural that software
engineers drool over the idea of being able to do this
themselves. Attempts to componentize software have been made and have
been partially successful. However, they have been successful at the
"uninteresting" level - the level of commodity software. In the early
days of operating systems, it made a real difference which operating
system you used because of all the different capabilities. However,
the current view is that it really doesn't matter too much what
operating system is abstracting away the hardware, so the interface
became less important as a competitive feature, making operating
system suppliers go away as an identifiable business and hardware
suppliers bundle some POSIX-compliant operating system with their
hardware.[5] The same holds for things like
databases, which do nothing else than implementing well-defined
behavior (a relational database) and most software developers don't
really care which one is storing tables and indexes from a technical
viewpoint. Typically, commodity software like operating systems and
databases is the first software to componentize - hide behind a
well-defined interface and abstract away implementation differences
for the user. Just as an average electronics engineer doesn't really care
whether a type 555 timer chip comes from Texas Instruments or NEC, an
average software engineer doesn't really care whether a database comes
from Oracle or Sybase. However, even for electronics the story doesn't
hold for higher-level components. I think the average electronics
engineer does care whether an embedded CPU comes from Zilog, Motorola
or Intel. So the analogy is correct in sofar that even if software can
be componentized as much as in electronics, this won't lead to an
automatic Nirvana because even in electronics, only commodity
components can be swapped freely, but key components
cannot. Furthermore, the implicit assumption that electronics
engineering is somehow easy doesn't hold as well, as we can witness in
the numerous hardware bugs that are popping up now that electronics
reaches previously unknown complexity levels (and starts to look more
and more like pure thought matter - software). I think, though, that the analogy is basically wrong in
that it makes a lot of assumptions about similarities between
electronic components interacting and software components
interacting. I think there's a difference in the kind of messages that
components send each other: in electronics, a lot of components are
"content" with not interpreting the data. There's a whole series of
components that operates on serial bit streams, bit by bit, doing
useful things to them on the higher (byte) level without knowledge of
that higher level. This makes interfaces simple: there's only one
"brand" of electric current, it is simple to agree on voltage and
current characteristics, and it is sufficient to do so. There aren't
many discrete components that would really perform better at 220V than
at TTL levels. And if there are, it is easy enough to build an adaptor
to hide these internal differences. Software components do normally interpret the data they
act on, apart from the lowest-level components. But the low-level
components of software are machine language statements (or maybe small
groups of them), not the complex entities we talk about when
discussing software components! A software component is very high
level, and has a very specific interface, just like in electronics
high-level components tend to have very specific interfaces (on
the level of software components, a television set consists of a cathode
ray tube, a high voltage amplifier, and a receiver. Apart from the
relatively simple CRT, I think it is probably easier to design amplifier and
receiver yourself than trying to bridge a Sony amplifier with a
Philips receiver "component"). Because messages between components are
very high-level and full of meaning and semantics, interfaces between
components become highly specialized and therefore automatically hard
to re-use in new, previously unthought-of situations. Furthermore,
there's much to gain functionality-wise and competition-wise from
altering and specializing these interfaces, so there's no internal
drive for standardization. There probably never was one in the
electronics supply side, but the fact that the demand side was
relatively small and well-organized (only the electronics industry)
made it easier for the demand side to require standardization. In fact, the only place where I can see that components
have worked in a grand way is the Unix operating system. By dumbing
down all data to character streams, it became possible to componentize
all kinds of specialized utilities that can operate on character
streams and let the user or administrator put these together in all kinds
of interesting and useful ways. I doubt, though, whether this is possible at
the level of strongly typed interfaces between objects as the current
mainstream idea seems to be.[6]
In the previous section, I have tried to show why all sorts of
analogies between software engineering and hardware engineering don't
hold. It is well-known that basing procedures and decisions on bad
analogies results in bad procedures and decisions, so I maintain that
it is not wise to try to base software engineering on analogies with
hardware engineering. The problem is, this rules out 99% of the existing software
engineering methodologies. Actually, this is not really a problem because most of these
methodologies don't work very well (apart probably for the couple that
do work like bridge building and do start with a couple of years of
design, testing and prototyping. So if you work for NASA working on
STS control software, you can stop reading here). The problem is,
of course, that
no-one seems to come up with something that does work very well, or
maybe that the ones that do come up with something don't do it from
the perspective of coming up with a methodology and therefore forget
to write a book about it. As I'll show later, these alternative
methodologies do exist but nobody calls them that way and "serious"
software shops don't use them. Let's look back at what a software
system is, so that we may come up with some perspective from first
principles. Most software can, or better: should, be viewed as software
systems consisting of a lot of elements in constant
interaction. In modern software technique, these elements are often
objects or constellations of objects, but even in earlier days people
often put block diagrams on the white board: "here's the spell-checker,
and this piece interprets files and converts them into the internal
data structure which is managed here and interpreted there for
display. Here's a piece that intercepts keystrokes and changes the
internal data structure, and finally there's the stuff that converts
the internal data structure to an external file." Each of these pieces
of a hypothetical word processor can be reduced to other pieces, and
if you break down often artificial hierarchical barriers you're left
with a sea of small pieces of code and data—objects,
subroutines, whatever—that interact with each other in
intricate, complex ways and which can and will adapt to each other
and the environment (by manual programming labor, but still). Not coincidentally, this is the same as the definition for a
complex adaptive system, which is seen as a
pre-requisite for a system that is capable of exhibiting complex,
emergent behavior. Where are the object models
and the design diagrams for the behavior that Life shows on the
screen? They aren't there, because the behavior emerged from the set
of rules that John Conway made. In the systems that complexity science
studies, there are no blueprints, only recipes[7]—whatever results from executing the
recipe cannot be predicted without actually executing the recipe (take
a cookbook and try it!), so the system's behavior is said to be
emergent, rather than pre-defined. Complex adaptive systems are said to live "on the edge of
chaos". They aren't linear, but they are not chaotic either. Because they are not
linear, you cannot readily predict what happens if you change a bit of
the system. Because elements of the system react to each other, and
because elements of the system never stabilize, the whole system is
constantly changing - the elements are adapting to environmental
changes as well as to their counterparts (the latter concept is called
coevolution in biology and it is regarded as a major driving force for
keeping evolution going on). There's another similarity between such systems and software
systems: all systems exhibiting complex adaptive behavior can be
abstracted away from the substrate and be put into the form of pure
information, like software. The intricate chemical interactions which
have probably lead to catalytic loops and ultimately to life; the not
much simpler system that is formed by a country's economy; all sorts
of ecological masterpieces of balance and arms race—ultimately,
they can be interpeted as software systems executing on various
types of computers. If we accept that software systems themselves fall under
this classification, we can explain a lot from it. The fact that bugs
occur (and are fixed, and result in other bugs); that it is hard to
predict what will happen when changing a component; that master
designs and detailed designs and whatever blueprints are made never
quite seem to match the actual software, so that they need to be
modified after the fact (of course, when this doesn't happen the designs
become obsolete merely by the fact that they are stable things in a
dynamic system). Furthermore, there is a lot to gain from this model:
if we accept it, it means that the only way to make software survive
is by constant change - by designing change into the system and allowing
behavior to emerge, it is possible to make sure that software
can match the pace of change that the outside worlds requires from
it. Furthermore, such software would be so adaptable that instead of
confronting every user with all functionality, or confronting every
user with the lowest common denominator, it could offer every user
with just the functionality he or she needs.[8]
A good software engineering approach would then be to stop with
engineering and start growing it. In fact, growing software
has been used successfully in what is now called the Open Source
movement but was formerly known as "spare time hacking". The prime
example (it is getting boring) is the Linux kernel. There is no master
plan, I have never seen a design document, there are just a whole bunch
of pieces of the kernel competing for survival, and a guy that plays
the role of "laissez-fair gardener": the garden just happens, anyone
can plant seeds or even whole plants, although
whatever the gardener regards as weed is mercilessly removed. The fun
thing about this kind of garden is, of course, that if a piece of weed
is removed, you can clone your own garden, grow the weed into a
beautiful flower, and then retry. The gardener, Linus Torvalds, just
defined the basic rules (some stuff about code formatting and dealing
with locks, timing and interrupts) and stood back. He planted the
seeds for what could become a complex adaptive system and that is what
happened. For example, he only implemented drivers for a
Minix-compatible file system himself. Soon (within the year), there
were the XIA file system, the Extended file-system, and not much later
the Second Extended file system competing for market share. The Second
Extended file system won, but as is usual in dynamic systems, this
victory is only momentarily: two modern journaling file systems are
already starting to enter the arena and surely one of them will be the
next temporary king-of-the-hill (or not, maybe it'll share the throne, who
knows). In the meantime, the
Minix file system fills a niche (it has a bit less overhead so it is
used for storage-tight applications like floppy disks) and the
Extended and XIA file systems have been deceased. By allowing and catering for competition, Linus has "grown" a
much better file system than if he would have sat behind his computer
and designed one himself. It is the natural way to get at good
software, and in contrast classic engineering methods seem a bit like
brute force: they get at a certain result, but often at the cost of
quality and adaptability. And change is what software is all about, so
the brute-forced systems simply lose because they cannot keep up with
the market pace.
So how to grow software? Well, two things are clear:
forget the design and instead focus on change. Grow software -
confront users early and often with changes, in fact make them used to
changes (personally, I think that users are far more capable in
dealing with software that accommodates itself to them on a daily
basis than to software that is completely redone to accommodate all
the users every year). Have a master gardener that defines the rules
and keeps things from going astray. Allow competing implementations
(several implementations of the same stuff for different users), and
make sure that users can select among them. A lot of this can be found
in the Extreme Programming methodology/philosophy, although as far as I can
see this system doesn't have an explicit complexity science background
but rather is a collection of best practice patterns. However, it is
probably about the only system that has at least the potential to
be adapted to some sort of explicit complex adaptive systems-based way of
building software, especially because it de-emphasizes design and puts
a lot of weight with refactoring, a process that is consistent with
growing software (refactoring can be seen as splitting plants,
removing extraneous branches, etcetera—basic garden
maintenance). It doesn't add competition between alternative
implementations, and this may well be the hardest part. Unix
file systems can be freely interchanged, because they all adhere to a
simple and well-defined interface (this is also a problem, because
lots of potentially interesting features cannot be implemented through
this interface); but how to inject useful competition into the
implementation of, say, an electronic mail system?
In this paper, I have tried to make clear why I think that
current software engineering practice doesn't match a very natural way
of looking at software systems, namely as complex adaptive
systems. From first principles, I have given some guidelines how
software development consistent with this view can be done, and
suggested Extreme Programming as a possible starting point for such a
process. [Brooks95] The Mythical Man-Month: Essays on Software Engineering, Frederick Brooks, 1995, 0-201-83595-9, Addison-Wesley. [Dawkins96] Climbing Mount Improbable, Richard Dawkins, 1996, Viking. [Gleick88] Chaos: The Making of a Science, James Gleick, 1988, William Heinemann. [Hofstadter79] Godel, Escher, Bach: An Eternal Golden Braid, Douglas Hofstadter, 1979, Basic Books. [Waldrop92] Complexity: The Emerging Science at the Edge of Order and Chaos, Mitchell Waldrop, 1992, Simon & Schuster. Notes
|
|
