Build || Search

On growing mattresses, the Library of Babel, DARPA and software configuration. How flexible systems collect complexity.

Internet shopping

To anyone that has ever done any internet shopping this thought might have occurred: “Does the internet offer every possible object for sale?“

To test this, think of a thing that might be purchased. Then try to find a site that sells what you imagined. Whether you came up with a neon yellow rubber chicken, a silver ring that tells time, or a plant that doubles as a pen holder, chances are you can find it. I think this illustrates how, given time and pressure, systems stretch to contain all realities possible within their boundaries.

Growing mattresses

Remove those boundaries and a new type of reality occurs. In the universe of The Hitchhiker’s Guide to the Galaxy you would not have to bother with internet shopping. You could steer your spaceship to the nearest planet that grows the object you are interested in, because:

“… in an infinitely large Universe such as, for instance, the one in which we live, most things one could possibly imagine, and a lot of things one would rather not, grow somewhere.”

If your ship gets decent mileage to the gallon this may work just as well, but you’ll need a good map. The problem has shifted from creating the right thing to finding the right thing. Or maybe that was the challenge all along.

The Library of Babel

An interesting middle ground between those two realities is described in Borges’ Library of Babel. The library contains all possible books of a defined length*:

“ … each bookshelf holds thirty-two books identical in format; each book contains four hundred ten pages; each page, forty lines; each line, approximately eighty black letters.”

“ For every rational line or forthright statement there are leagues of senseless cacophony, …”

This is a system with clear, hard constraints, still enveloping such a maddening amount of possibilities that writing the complete works of Shakespeare would be less work than finding them.

System yoga

Remember when you first got that business requirement that the system you were about to build had to be “flexible”? I’m sure you also went back to your desk scratching your head about that one. I generally ended up adding some extra configuration options and left it at that.

Years later, when consultants came pitching their systems as flexible, I got suspicious.

Engineering time is a scarce resource in almost any company. And that’s a good thing. It forces us to think about what we need most urgently and helps prevent the business wasting time and money implementing trivial features. It also makes flexible configuration seem like a good idea. We can now change system behavior without spending valuable engineering time. The problem is though, … we will.

Config only

The more configurable a system is, the more possible realities it can contain. Lack of engineering time creates the right pressure to configure the system in surprising ways. Given enough time it will absorb realities that no one dreamt of when building it. Just like DARPA was unlikely to imagine that the resilient communication network they were building would become so useful to sell fluffy rainbow unicorn slippers. (Yes, at least 40 options available for purchase).

“Highly configurable” is a polite term for carelessly constrained. A pocket of chaos waiting to emerge. Soon the code cannot be safely changed because creative users have configured the system in unexpected ways. The system’s behavior can no longer be understood through its code, so it can not be understood at all. This makes any kind of change a hazardous adventure.

Building or searching

We like to say we build software, create architectures, lay foundations. Maybe building is an appealing analogy because it is associated with physical architecture, our most durable cultural expressions. But is it accurate?

Because Borges’ library and the Hitchhikers universe already contain all their possibilities the dominant problem becomes search. When we think of creating software as implementing the constraints that best model the business reality, that reality also already exists in full. In a complex business a big part of the development effort becomes discovery and analysis. Searching for the rules that will limit the complexity the system can contain. This search can easily take more time and effort than implementing the construct that will hold those constraints.

We need to search the chaotic reality carefully to find the patterns we need to include. The deeper the search, the stronger the constraints can be. That does not always mean adding more detail. Often it is better to make hard choices about what edge cases are not relevant enough to support.

The quality of the search largely determines the quality of the system.

So maybe the core problem is search after all.

* Although in the story the library is described as “perhaps infinite”, the amount of possible books it can contain is not. The constraints applied by Borges limit the library to a large, but finite number of unique possibilities.