The trick is to realize that there"s no real contradiction here. You want to be optimistic and skeptical about two different things. You have to be optimistic about the possibility of solving the problem, but skeptical about the value of whatever solution you"ve got so far.

People who do good work often think that whatever they"re working on is no good. Others see what they"ve done and think it"s wonderful, but the creator sees nothing but flaws. This pattern is no coincidence: worry made the work good.

If you can keep hope and worry balanced, they will drive a project forward the same way your two legs drive a bicycle forward. In the first phase of the twocycle innovation engine, you work furiously on some problem, inspired by your confidence that you"ll be able to solve it. In the second phase, you look at what you"ve done in the cold light of morning, and see all its flaws very clearly. But as long as your critical spirit doesn"t outweigh your hope, you"ll be able to look at your admittedly incomplete system and think, how hard can it be to get the rest of the way?

It"s tricky to keep the two forces balanced. In young hackers, optimism predominates. They produce something, are convinced it"s great, and never improve it. In old hackers, skepticism predominates, and they won"t even dare to take on ambitious projects.

Anything you can do to keep the redesign cycle going is good. Prose can be rewritten over and over until you"re happy with it. But software, as a rule, doesn"t get redesigned enough. Prose has readers, but software has users. If a writer rewrites an essay, people who read the new version are unlikely to complain that their thoughts have been broken by some newly introduced incompatibility.

Users are a double-edged sword. They can help you improve your language, but they can also deter you from improving it. So choose your users carefully, and be slow to grow their number. Having users is like optimization: the wise course is to delay it. Also, as a general rule, you can at any given time get away with changing more than you think. Introducing change is like pulling off a bandage: the pain is a memory almost as soon as you feel it.

Everyone knows it"s not a good idea to have a language designed by a committee. Committees yield bad design. But I think the worst danger of committees is that they interfere with redesign . It"s so much work to introduce changes that no one wants to bother. Whatever a committee decides tends to stay that way, even if most of the members don"t like it.

Even a committee of two gets in the way of redesign. This happens particularly in the interfaces between pieces of software written by two different people. To change the interface both have to agree to change it at once. And so interfaces tend not to change at all, which is a problem because they tend to be one of the most ad hoc parts of any system.

One solution here might be to design systems so that interfaces are horizontal instead of vertical-so that modules are always vertically stacked strata of abstraction. Then the interface will tend to be owned by one of them. The lower of two levels will either be a language in which the upper is written, in which case the lower level will own the interface, or it will be a slave, in which case the interface can be dictated by the upper level.

14.10. The Dream Language

By way of summary, let"s try describing the hacker"s dream language. The dream language is clean and terse. It has an interactive top level that starts up fast. You can write programs to solve common problems with very little code. Nearly all the code in any program you write is code that"s specific to your application. Everything else has been done for you.

The syntax of the language is brief to a fault. You never have to type an unnecessary character, or even use the Shift key much.

Using big abstractions you can write the first version of a program very quickly. Later, when you want to optimize, there"s a really good profiler that tells you where to focus your attention. You can make inner loops blindingly fast, even writing inline byte code if you need to.

There are lots of good examples to learn from, and the language is intuitive enough that you can learn how to use it from examples in a couple minutes. You don"t need to look in the manual much. The manual is thin, and has few warnings and qualifications.

The language has a small core, and powerful, highly orthogonal libraries that are as carefully designed as the core language. The libraries all work well together; everything in the language fits together like the parts in a fine camera. Nothing is deprecated or retained for compatibility. The source code of all the libraries is readily available. It"s easy to talk to the operating system and to applications written in other languages.

The language is built in layers. The higher-level abstractions are built in a transparent way out of lower-level abstractions, which you can get hold of if you want.

Nothing is hidden from you that doesn"t absolutely have to be. The language offers abstractions only as a way of saving you work, rather than as a way of telling you what to do. In fact, the language encourages you to be an equal partic.i.p.ant in its design. You can change everything about it, including even its syntax, and anything you write has, as much as possible, the same status as what comes predefined. The dream language is not only open source, but open design.

Chapter 15. Design and Research.

Visitors to this country are often surprised to find that Americans like to begin a conversation by asking "what do you do?" I"ve never liked this question. I"ve rarely had a neat answer to it. But I think I have finally solved the problem. Now, when someone asks me what I do, I look them straight in the eye and say, "I"m designing a new dialect of Lisp." I recommend this answer to anyone who doesn"t like being asked what they do. The conversation will turn immediately to other topics.

I don"t consider myself to be doing research on programming languages. I"m just designing one, in the same way that someone might design a building or a chair or a new typeface. I"m not trying to discover anything new. I just want to make a language that will be good to program in.

The difference between design and research seems to be a question of new versus good. Design doesn"t have to be new, but it has to be good. Research doesn"t have to be good, but it has to be new. I think these two paths converge at the top: the best design surpa.s.ses its predecessors by using new ideas, and the best research solves problems that are not only new, but worth solving. So ultimately design and research are aiming for the same destination, just approaching it from different directions.

What do you do differently when you treat programming languages as a design problem instead of a research topic?

The biggest difference is that you focus more on the user. Design begins by asking, who is this for and what do they need from it? A good architect, for example, does not begin by creating a design that he then imposes on the users, but by studying the intended users and figuring out what they need.

Notice I said "what they need," not "what they want." I don"t mean to give the impression that working as a designer means working as a sort of short-order cook, making whatever the client tells you to. This varies from field to field in the arts, but I don"t think there is any field in which the best work is done by the people who just make exactly what the customers tell them to.

The customer is always right in the sense that the measure of good design is how well it works for the user. If you make a novel that bores everyone, or a chair that"s horribly uncomfortable to sit in, then you"ve done a bad job, period. It"s no defense to say that the novel or chair is designed according to the most advanced theoretical principles.

And yet, making what works for the user doesn"t mean simply making what the user tells you to. Users don"t know what all the choices are, and are often mistaken about what they really want. It"s like being a doctor. You can"t just treat a patient"s symptoms. When a patient tells you his symptoms, you have to figure out what"s actually wrong with him, and treat that.

This focus on the user is a kind of axiom from which most of the practice of good design can be derived, and around which most design issues center.

When I say that design must be for users, I don"t mean to imply that good design aims at some kind of lowest common denominator. You can pick any group of users you want. If you"re designing a tool, for example, you can design it for anyone from beginners to experts, and what"s good design for one group might be bad for another. The point is, you have to pick some group of users. I don"t think you can even talk about good or bad design except with reference to some intended user.

You"re most likely to get good design if the intended users include the designer himself. When you design something for a group that doesn"t include you, it tends to be for people you consider less sophisticated than you, not more sophisticated. And looking down on the user, however benevolently, always seems to corrupt the designer. I suspect few housing projects in the US were designed by architects who expected to live in them. You see the same thing in programming languages. C, Lisp, and Smalltalk were created for their own designers to use. Cobol, Ada, and Java were created for other people to use.

If you think you"re designing something for idiots, odds are you"re not designing something good, even for idiots.

Even if you"re designing something for the most sophisticated users, though, you"re still designing for humans. It"s different in research. In math you don"t choose abstractions because they"re easy for humans to understand; you choose whichever make the proof shorter. I think this is true for the sciences generally. Scientific ideas are not meant to be ergonomic.

Over in the arts, things are different. Design is all about people. The human body is a strange thing, but when you"re designing a chair, that"s what you"re designing for, and there"s no way around it. All the arts have to pander to the interests and limitations of humans. In painting, for example, all other things being equal a painting with people in it will be more interesting than one without. It is not merely an accident of history that the great paintings of the Renaissance are all full of people. If they hadn"t been, painting as a medium wouldn"t have the prestige it does.

Like it or not, programming languages are also for people, and I suspect the human brain is just as lumpy and idiosyncratic as the human body. Some ideas are easy for people to grasp and some aren"t. For example, we seem to have a very limited capacity for dealing with detail. It"s this fact that makes programming languages a good idea in the first place; if we could handle the detail, we could just program in machine language.

Remember, too, that languages are not primarily a form for finished programs, but something that programs have to be developed in. Anyone in the arts could tell you that you might want different mediums for the two situations. Marble, for example, is a nice, durable medium for finished ideas, but a hopelessly inflexible one for developing new ideas.

A program, like a proof, is a pruned version of a tree that in the past has had false starts branching off all over it. So the test of a language is not simply how clean the finished program looks in it, but how clean the path to the finished program was. A design choice that gives you elegant finished programs may not give you an elegant design process. For example, I"ve written a few macro defining macros that look now like little gems, but writing them took hours of the ugliest trial and error, and frankly, I"m still not entirely sure they"re correct.

We often act as if the test of a language were how good finished programs look in it. It seems so convincing when you see the same program written in two languages, and one version is much shorter. When you approach the problem from the direction of the arts, you"re less likely to depend on this sort of test. You don"t want to end up with a programming language like marble.

For example, it is a huge win in developing software to have an interactive toplevel, what in Lisp is called a read-eval-print loop. And when you have one, this has real effects on the design of the language. It would not work well for a language where you have to declare variables before using them. When you"re just typing expressions into the toplevel, you want to be able to set x to some value and then start doing things to x. You don"t want to have to declare the type of x first. You may dispute either of the premises, but if a language has to have a toplevel to be convenient, and mandatory type declarations are incompatible with a toplevel, then no language that makes type declarations mandatory could be convenient to program in.

To get good design you have to get close, and stay close, to your users. You have to calibrate your ideas on actual users constantly. One of the reasons Jane Austen"s novels are so good is that she read them out loud to her family. That"s why she never sinks into self-indulgently arty descriptions of landscapes, or pretentious philosophizing. (The philosophy"s there, but it"s woven into the story instead of being pasted onto it like a label.) If you open an average "literary" novel and imagine reading it out loud to your friends as something you"d written, you"ll feel all too keenly what an imposition that kind of thing is upon the reader.

In the software world, this idea is known as Worse is Better. Actually, there are several ideas mixed together in the concept of Worse is Better, which is why people are still arguing about whether worse is actually better or not. But one of the main ideas in that mix is that if you"re building something new, you should get a prototype in front of users as soon as possible.

The alternative approach might be called the Hail Mary strategy. Instead of getting a prototype out quickly and gradually refining it, you try to create the complete, finished product in one long touchdown pa.s.s. Countless startups destroyed themselves this way during the Internet Bubble. I"ve never heard of a case where it worked.

What people outside the software world may not realize is that Worse is Better is found throughout the arts. In drawing, for example, the idea was discovered during the Renaissance. Now almost every drawing teacher will tell you that the right way to get an accurate drawing is not to work your way slowly around the contour of an object, because errors will acc.u.mulate and you"ll find at the end that the lines don"t meet. Instead you should draw a few quick lines in roughly the right place, and then gradually refine this initial sketch.

In most fields, prototypes have traditionally been made out of different materials. Typefaces to be cut in metal were initially designed with a brush on paper. Statues to be cast in bronze were modelled in wax. Patterns to be embroidered on tapestries were drawn on paper with ink wash. Buildings to be constructed from stone were tested on a smaller scale in wood.

What made oil paint so exciting, when it first became popular in the fifteenth century, was that you could make the finished work from the prototype. You could make a preliminary drawing if you wanted to, but you weren"t held to it; you could work out all the details, and even make major changes, as you finished the painting.

You can do this in software too. A prototype doesn"t have to be just a model; you can refine it into the finished product. I think you should always do this when you can. It lets you take advantage of new insights you have along the way. But perhaps even more important, it"s good for morale.

Morale is key in design. I"m surprised people don"t talk more about it. One of my first drawing teachers told me: if you"re bored when you"re drawing something, the drawing will look boring. For example, suppose you have to draw a building, and you decide to draw each brick individually. You can do this if you want, but if you get bored halfway through and start making the bricks mechanically instead of observing each one, the drawing will look worse than if you had merely suggested the bricks.

Building something by gradually refining a prototype is good for morale because it keeps you engaged. In software, my rule is: always have working code. If you"re writing something you"ll be able to test in an hour, you have the prospect of an immediate reward to motivate you. The same is true in the arts, and particularly in oil painting. Most painters start with a blurry sketch and gradually refine it. If you work this way, then in principle you never have to end the day with something that looks unfinished. Indeed, there is even a saying among painters: "A painting is never finished. You just stop working on it." This idea will be familiar to anyone who has worked on software.

Morale is another reason that it"s hard to design something for an unsophisticated user. It"s hard to stay interested in something you don"t like yourself. To make something good, you have to be thinking, "wow, this is really great," not "what a piece of s.h.i.t; those fools will love it."

Design means making things for humans. But it"s not just the user who"s human. The designer is human too.