Is is preferable to design top down or bottom up?

Top-down is a great way to describe things you know, or to re-build things that you’ve already built.

Top-down biggest problem is that quite often simply there is no “top”. You will change your mind about what the system should do while developing the system and while exploring the domain. How can be your starting point something that you don’t know (i.e. what you want the system to do)?

A “local” top down is a good thing… some thinking ahead of coding is clearly good. But thinking and planning too much is not, because what you are envisioning is not the real scenario (unless you’ve already been there before, i.e. if you are not building, but re-building). Global top-down when building new things is just nonsense.

Bottom-up should be (globally) the approach unless you know 100% of the problem, you need just the known solution to be coded and you don’t care about looking for possible alternative solutions.

Lisp approach is the distilled bottom-up. You not only build bottom up but you can also shape the bricks the way you need them to be. Nothing is fixed, freedom is total. Of course freedom takes responsibility and you can make horrible things by misusing this power.

But horrible code can be written in any language. Even in languages that are shaped as cages for the mind, designed with the hope that with those languages even monkeys could get good programs up and running (an idea so wrong on so many levels that it hurts even just thinking about it).

Your example is about a web server. Now in 2012 this is a well-defined problem, you have specs to be followed. A web server is just an implementation problem.
Especially if you are aiming at writing a web server substantially identical to the other gajillion of web servers that are out there then nothing is really unclear, except some minutiae. Even your comment about RSA is still talking about a clearly defined problem, with formal specifications.

With a well defined problem, with formal specifications and already known solutions then coding is just connecting in the dots. Top down is ok for that. This is the project manager heaven.

In many cases however there is no proven well-known approach to be used to connect the dots. Actually very often is hard to say even what are the dots.

Suppose for example you are asked to instruct an automatic cutting machine to align the parts to be cut to a printed material that is not perfectly conforming to the theoretic repetitive logo. You are given the parts and pictures of the material as taken by the machine.

What is an alignment rule? You decide. What is a pattern, how to represent it? You decide. How to align the parts? You decide. Can parts be “bent”? It depends, some not and some yes, but of course not too much. What to do if the material is just too deformed for a part to cut it acceptably? You decide. Are all the material rolls identical? Of course not, but you cannot bug the user to adapt alignment rules for every roll… that would be impractical. What pictures are seeing the cameras? The material, whatever that may mean… it can be color, it can be black over black where just the light reflex makes the pattern evident. What does it mean to recognize a pattern? You decide.

Now try to design the general structure of a solution for this problem and give a quote, in money and time. My bet is that even your system architecture… (yes, the architecture) will be wrong. Cost and time estimation will be random numbers.

We implemented it and now it’s a working system, but changed our mind about the very shape of the system a big number of times. We added entire sub-systems that now cannot even be reached from the menus. We switched master/slave roles in protocols more than once. Probably now we’ve enough knowledge to attempt re-building it better.

Other companies of course did solve the same problem… but unless you are in one of these companies most probably your top-down detailed project will be a joke. We can design it top-down. You cannot because you never did it before.

You can probably solve the same problem too. Working bottom-up however. Starting with what you know, learning what you don’t and adding up.

New complex software systems are grown, not designed. Every now and then someone starts designing a big new complex ill-specified software system from scratch (note that with a big complex software project there are only three possibilities: a] the specification is fuzzy, b] the specification is wrong and self-contradictory or c] both… and most often [c] is the case).

These are the typical huge-company projects with thousands and thousands of hours thrown into powerpoint slides and UML diagrams alone. They invariably fail completely after burning embarrassing amounts of resources… or in some very exceptional case they finally deliver an overpriced piece of software that implements only a tiny part of the initial specs. And that software invariably is deeply hated by users… not the kind of software you would buy, but the kind of software you use because you’re forced to.

Does this mean that I think that you should think only to code? Of course not. But in my opinion the construction should start from bottom (bricks, concrete code) and should go up… and your focus and attention to detail should in a sense “fade” as you are getting farther from what you have. Top-down is often presented as if you should put the same level of detail to the whole system at once: just keep it splitting every node until everything is just obvious… in reality modules, subsystem are “grown” from subroutines.
If you do not have a previous experience in the specific problem your top down design of a subsystem, module or library will be horrible. You can design a good library once you know what functions to put in, not the other way around.

Many of the Lisp ideas are getting more popular (first class functions, closures, dynamic typing as default, garbage collection, metaprogramming, interactive development) but Lisp is still today (among the languages I know) quite unique in how easy is to shape code for what you need.

Keyword parameters for example are already present, but if they were not present they could be added. I did it (including keyword verification at compile time) for a toy Lisp compiler I am experimenting with and it doesn’t take much code.

With C++ instead the most you can get is a bunch of C++ experts telling you that keyword parameters are not that useful, or an incredibly complex, broken, half backed template implementation that indeed is not that useful.
Are C++ classes first-class objects? No and there’s nothing you can do about it. Can you have introspection at runtime or at compile time? No and there’s nothing you can do about it.

This language flexibility of Lisp is what makes it great for bottom-up building. You can build not only subroutines, but also the syntax and the semantic of the language. And in a sense Lisp itself is bottom-up.