Note: is in rebuild. Please accept my apologies for broken links, missing stuff, etcetera - more

Art, Science, Engineering, Craft

Art, Science, Engineering, Craft

Approaches to Building Software

Cees de Groot

This paper examines various approaches to building software, and tries to put each one into its right place. The central argument is that there is no single "correct" approach.

Table of Contents
1. Introduction
2. Art
3. Craft
4. Engineering
5. Science
6. Conclusion

1. Introduction

Time and time again, flamewars rage over the Usenet concerning the topic of whether building software is an art, a science, an engineering discipline, or a craft. It is a very interesting topic, not in the last place because one possible outcome of the discussion is whether people who build software must be accredited engineers - at the moment, various institutions in the United States are working on the subject of certified Software Engineer, to be put at the same level as other Engineers.

One's view towards building software largely influences one's view towards the methodology of building software. Lightweight methodologies seem to emphasize the craft aspect; work in logic and formal methods emphasizes the science aspect; while heavyweight methodologies stress the engineering side. Furthermore, it can be argued that the great admiration for certain individuals in the Open Source movement is explainable by the view of (mostly younger) programmers of software building as a form of art.

In order to structure the discussion, therefore, I want to use this paper as an inventorying excercise. I am known to hold fairly strong opinions on what I think is "right", but for in this particular instance I will try to stay neutral. The four aspects will be discussed in alphabetic order - I think it is shortsighted to rank a particular one as being "less" than any other one. People who do this don't deserve to be part of this very important discussion about the basics of our profession. As it happens, the order makes up for a good flow of arguments, but this is pure coincidence.

2. Art

Art in the meaning we will use here is defined by Merriam-Webster OnLine as "the conscious use of skill and creative imagination especially in the production of aesthetic objects".

In the definition, there are two things that set Art apart from the others: 1) a stress of creativity, and 2) the fact that the product is something of mainly aesthetic value.

That software can be used to create art is evident: from the earliest days of computers on, programmers have applied their programming skills and their creativity in order to produce software that created "aesthetic objects". In the early days, line printers were coerced into producing pictures built up from lots of different characters; soon after the introduction of multi-media capable microcomputers highly skilled programmers started bringing out "demos", intricately crafted presentations pushing the computer to its limits. ACM's Siggraph announcements are customarily illustrated with both scientific and artistic imagery.

However, for the purpose of this paper I think it is more correct to view software, or source code, as the medium, not as the tool. The question then becomes one of a manner of building programs mainly for the aesthetic value of the source code.

Since 1984, a yearly competition deals with exactly that: source code mainly written for aesthetic value. The International Obfuscated C Code Contest gives prizes to entrants who come up with novel or interesting (or plain disgusting ways) to implement algorithms in the C programming language. More informally, programmers often code up algorithms in novel ways for "hack value", up to the point where simple operations that normally run in constant time are implemented in crazy algorithms that require polynomial execution time.

Therefore, writing software can be an art form, both with software being the tool as well as the medium. Needless to say, the market for the latter category will be quite small (only colleague programmers will be able to read the code, and often you need to be quite an expert to fully understand all the details). Furthermore, programming as an art, thus mainly for aesthetics, suffers the same drawback as other artforms: it is nigh impossible to make a living off it, and especially for the "art with software as the medium" category it is probably impossible to find a Maecenas.

3. Craft

Merriam-Webster OnLine defines Craft as "an occupation or trade requiring manual dexterity or artistic skill".

There's an interesting circularity here: the definition of Art refers to skill (something typically associated with Craft), while the definition of Craft refers back to "artistic skill". A liberal interpretation here would be that Art is typically associated with production for aesthetic value, while Craft is associated with production for use value, but they largely pull from the same set of skills (and creativity).

My personal experience with woodworking, music and photography seems to confirm this view. I learnt woodworking when I was boatsman for our student rowing club, from an old master who could put a competition skiff together from mahogany triplex plates, balsa wood, and his hands. Lots of the tools were made by his father, and I think it was typical for the craft aspect that design drawings were sketchy (in the literal sense). Photography sees a similar debate as software: some see it as a pure artform, others as a craft. Because, as a process, is applied physics and chemistry, one could even hold the viewpoint that it is engineering, and indeed some photographer's darkrooms and studios are full with measuring equipment. Finally, music (in my case: piano and trombone) is generally seen as an art form, although my mother often wondered what Czerny Etudes have to do with art...

All three forms of expression have a similar basis: in order to master them, you need a lot of skill. For woodworking, this means practicing on scrap wood; for photography, it means lots of test photos; for music, it means lots of etudes. Additionally, knowledge is important: you need to know about properties of wood, about the differences between various chrome emulsions, about music history for correct interpretation. The groundwork, therefore, for Art and Craft, is the same: practice the skill, read about it, and have a master who can correct and guide you.

The deviation becomes apparent only when you start considering the goal. As we've seen, for Art this goal is mostly aesthetics. Craft puts the emphasis onto use, which constrains the amount of creativity you can put into it. A craftsman often gets commissioned work, and you don't have a long career if you build boats with beautiful holes in them or take pictures of just the bride, because you think the groom is ugly and he would spoil the aesthetic value of your photographs.

Software as a craft is practiced especially in smaller shops, which shouldn't surprise us because this parallels other disciplines (the sole reason that the car industry is what it is today is that Ford did away with craftsmen). The work is typically done in the context of light-weight methodologies, because small shops emphasize interpersonal communication which is more efficient than lots of paperwork but doesn't scale well to larger teams.

I don't have the exact figures, but my guess is that small teams (lets arbitrarily say this is less than 25 people) are responsible for more than 90% of all software written. Don't forget that software systems within large corporations are often written by small teams inside the enterprise; even when they are part of the larger entity, if the system is insulated well enough from other teams' efforts, the team can be regarded as a small software shop.

A small team typically has total commitment to the end product (the source code), and cares less for lots of drawings, formality, etcetera. This is very much the way a craftsman goes about - he knows his skills, the medium he works in, and therefore can set about executing his assignment without too much research and ritual. The fact that he has a fairly good idea where things can go wrong helps him to go slower at tricky points, make sure that there's a way back (for example, a woodsworker may first fit everything together before applying glue - if parts don't fit, she knows that you can just throw away the single wrong part instead of half the work).

Building software is supportive of this approach, because it is so infinitely fluid. Everything can always be changed, thrown away, redone, until it is just right. Writing and composing share this property, and any woodworker (let alone any sculptor) is probably very jealous of writers, composers, and programmers!

4. Engineering

Engineering is defined by Merriam-Webster OnLine as "the application of science and mathematics by which the properties of matter and the sources of energy in nature are made useful to people".

For the purposes of this discussion, I think it is useful to emphasize the result of engineering: it helps us scale beyond what Craft can produce. In many cases, the application of "science and mathematics" in engineering didn't help humanity by producing new things, but by making the product available to many of us. Indeed, engineers are often the ones who translate findings of science (or "craftsmen" - the proverbial lone inventors probably qualify for that title) into mass production methods.

Engineers succeed in their discipline by being very rigorous. Where a craftsman can wander about, probe, stumble, and still put a high quality product together, an engineer does not have this luxury: a production line cannot afford to experiment about; the thing either works or it is a failure. Rigor and scalability come at a price, however: lots of it is attained by replacing informal methods with (often expensive) formal methods, and by replacing inter-person communication with volumes of written documentation: procedures, standards, designs, plans, etcetera.

Clearly engineering is an astounding success, as probably 90% of anything physical you posess has been engineered, and not crafted. There is a price to pay, of course. When an engineer makes a mistake, it multiplies by the number of units produced. And a table that the local carpenter built for you has a different feel than the one you got from Ikea. Engineering emphasizes utility, which is probably something else than "use", the word I applied to Craft. The difference, I think, is the Alexandrian "Quality Without a Name"

It is no surprise that "software engineering" is a dream of many in the field. It bears the promises of measurable quality, methodical approaches, and a way out of the "software crisis". Large teams can indeed benefit from approaches borrowed from other engineering practices, because inter-person communication doesn't scale well, and above a certain size procedures and written documentation become inevitable ways for team members to coordinate.[1] Large software teams subscribe to heavyweight methodologies, which are probably very much the same as the methodologies applied in other engineering fields: documents are produced, verified for correctness and quality, and passed on. Whereas the design of small "craft" teams is hidden in the source code, the design of large "engineering" teams is hidden in the documentation.

4.1. Public safety

I discussed engineering as a matter of scalability, but the second well-known aspect is the matter of public safety. Licensed engineers are educated in such a way that they are very concerned about the effects of their actions on public safety. It may be economic to leave shielding from wires, but public safety certainly isn't furthered by it.

Software builders, especially those who work in public safety-critical areas, probably can learn a lot from these engineers. I think it is a separate track of the "is software engineering" issue, one that is - to an extent - orthogonal to the scalability feature of enginnering

5. Science

In Merriam-Webster OnLine, Science is defined as "the state of knowing: knowledge as distinguished from ignorance or misunderstanding". Another meaning probably also applies: "3a: knowledge or a system of knowledge covering general truths or the operation of general laws especially as obtained and tested through scientific method".

Science "feeds" Engineering, and to a lesser extend Craft and Art as well. Just as electronics engineers don't run away with every research paper that physicists publish, I think it is prudent for software builders to be selective in the application of computer science research results to the daily routine. There is a practicality test to be made and history has shown that this filtering function can only be executed by workers in the field, not by the scientists themselves (often to the frustration of the latter group, who cannot believe why people keep using "old" technologies). There are a lot of debates going on between computer scientists and field workers, and they do or do not influence the field (object-oriented programming conquered the field, but functional programming still seems to have problems catching on). This relation with science, however, doesn't make the practice of building software a science in itself.

Software can be part of science, of course. There is academic research, both in the field of computer science as in the field of pure mathematics (in a sense, software is applied mathematics and software can be described in pure mathematical terms). Computer science research seems to be quite healthy in that it seems to fill a bell curve: a lot of "standard" stuff in the middle - performance tuning, tweaks on existing systems, etcetera; a bit of completely uninteresting or plain wrong work (I'll refrain from giving examples here), and a bit of highly interesting, innovative research (over the years: things like OO programming, RISC chips, functional languages, and so on).

Science goes one step further than engineering in rigor. It is often not enough to show that something is working (by duly testing it), but the requirement is to prove why something is working as it works, and describe it in such a way to make the work repeatable. This calls for extraordinary formalism, which in itself is an interesting research area (proof-carrying code, for example). Some engineering shops probably can benefit from these approaches, like those who help building nuclear plants or large passenger aircraft. Proving that something works is possible, and a scientific proof that withstands scrutiny of peers after publication in a journal is probably the best way to make sure that the quality is at the highest standards.

6. Conclusion

To be written after initial comments.



One of my basic beliefs is that even with very thorough formal methods, the medium of software is so young and so badly understood that it is very risky to build large teams at all, but I promised a neutral paper.

Copyright (C)2000-2011 Cees de Groot -- All rights reserved.