My book, Designing Data-Intensive Applications, was published by O’Reilly in March 2017.
Published by Martin Kleppmann on 08 Oct 2012.
The problem of overly complex software is nothing new; it is almost as old as software itself. Over and over again, software systems become so complex that they become very difficult to maintain and very time-consuming and expensive to modify. Most developers hate working on such systems, yet nevertheless we keep creating new, overly complex systems all the time.
Much has been written about this, including classic papers by Fred Brooks (No Silver Bullet), and Ben Moseley and Peter Marks (Out of the Tar Pit). They are much more worth reading than this post, and it is presumptuous of me to think I could add anything significant to this debate. But I will try nevertheless.
Pretty much everyone agrees that if you have a choice between a simpler software design and a more complex design, all else being equal, that simpler is better. It is also widely thought to be worthwhile to deliberately invest in simplicity — for example, to spend effort refactoring existing code into a cleaner design — because the one-off cost of refactoring today is easily offset by the benefits of easier maintenance tomorrow. Also, much thought by many smart people has gone into finding ways of breaking down complex systems into manageable parts with manageable dependencies. I don’t wish to dispute any of that.
But there is a subtlety that I have been missing in discussions about software complexity, that I feel somewhat ambivalent about, and that I think is worth discussing. It concerns the points where external humans (people outside of the team maintaining the system) touch the system — as developers using an API exposed by the system, or as end users interacting with a user interface. I will concentrate mostly on user interfaces, but much of this discussion applies to APIs too.
Let me first give a few examples, and then try to extract a pattern from them. They are examples of situations where, if you want, you can go to substantial engineering effort in order to make a user interface a little bit nicer. (Each example based on a true story!)
ifstatements based on the order parameters. So you go and build a natural language grammar model for constructing emails based on sentence snippets (providing pluralisation, agreement, declension in languages that have it, etc), in such a way that for any one out of 100 million possible parameter combinations, the resulting email is grammatically correct and easy to understand.
We said above that simplicity is good. However, taking simplicity to an exaggerated extreme, you end up with software that does nothing. This implies that there are aspects of software complexity that are essential to the user’s problem that is being solved. (Note that I don’t mean complexity of the user interface, but complexity of the actual code that implements the solution to the user’s problem.)
Unfortunately, there is a lot of additional complexity introduced by stuff that is not directly visible or useful to users: stuff that is only required to “grease the wheels”, for example to make legacy components work or to improve performance. Moseley and Marks call this latter type accidental complexity, and argue that it should be removed or abstracted away as much as possible. (Other authors define essential and accidental complexity slightly differently, but the exact definition is not important for the purpose of this post.)
This suggests that it is important to understand what user problem is being solved, and that’s where things start getting tricky. When you say that something is essential because it fulfils a user requirement (as opposed to an implementation constraint or a performance optimisation), that presupposes a very utilitarian view of software. It assumes that the user is trying to get a job done, and that they are a rational actor. But what if, say, you are taking an emotional approach and optimising for user delight?
What if the user didn’t know they had a problem, but you solve it anyway? If you introduce complexity in the system for the sake of making things a little nicer for the user (but without providing new core functionality), is that complexity really essential? What if you add a little detail that is surprising but delightful?
You can try to reduce an emotional decision down to a rational one — for example, you can say that when a user plays a game, it is solving the user’s problem of boredom by providing distraction. Thus any feature which substantially contributes towards alleviating boredom may be considered essential. Such reductionism can sometimes provide useful angles of insight, but I think a lot would be lost by ignoring the emotional angle.
You can state categorically that “great user experience is an essential feature”. But what does that mean? By itself, that statement is so general that could be used to argue for anything or nothing. User experience is subjective. What’s preferable for one user may be an annoyance for another user, even if both users are in the application’s target segment. Sometimes it just comes down to taste or fashion. User experience tends to have an emotional angle that makes it hard to fit into a rational reasoning framework.
What I am trying to get at: there are things in software that introduce a lot of complexity (and that we should consequently be wary of), and that can’t be directly mapped to a bullet point on a list of user requirements, but that are nevertheless important and valuable. These things do not necessarily provide important functionality, but they contribute to how the user feels about the application. Their effect may be invisible or subconscious, but that doesn’t make them any less essential.
Returning to the examples above: as an application developer, you can choose whether to take on substantial additional complexity in the software in order to simplify or improve the experience for the user. The increased software complexity actually reduces the complexity from the user’s point of view. These examples also illustrate how user experience concerns are not just a matter of graphic design, but can also have a big impact on how things are engineered.
The features described above arguably do not contribute to the utility of the software — in the e-commerce example, orders will be fulfilled whether or not the confirmation emails are grammatical. In that sense, the complexity is unnecessary. But I would argue that these kind of user experience improvements are just as important as the utility of the product, because they determine how users feel about it. And how they feel ultimately determines whether they come back, and thus the success or failure of the product.
One could even argue that the utility of a product is a subset of its user experience: if the software doesn’t do the job that it’s supposed to, then that’s one way of creating a pretty bad experience; however, there are also many other ways of creating a bad experience, while remaining fully functional from a utilitarian point of view.
The emotional side of user experience can be a difficult thing for organisations to grapple with, because it doesn’t easily map to metrics. You can measure things like how long a user stayed on your site, how many things they clicked on, conversion rates, funnels, repeat purchase rates, lifetime values… but those numbers tell you very little about how happy you made a user. So you can take a “data-driven” approach to design decisions and say that a feature is worthwhile if and only if it makes the metrics go up — but I fear that an important side of the story is missed if you go solely by the numbers.
This is as far as my thinking has got: believing that a great user experience is essential for many products; and recognising that building a great UX is hard, can require substantial additional complexity in engineering, and can be hard to justify in terms of logical arguments and metrics. Which leaves me with some unanswered questions:
I suspect that the answers to the questions above are “it depends”. More precisely, “how one thing is valued relative to another is an aspect of your particular organisation’s culture, and there’s no one right answer”. That would imply that each of us should think about it; you should have your own personal answers for how you decide these things in your own projects, and be able to articulate them. But it’s difficult — I don’t think hard-and-fast rules have a chance of working here.
I’d love to hear your thoughts in the comments below. If you liked this post, you can subscribe to email notifications when I write something new :)