Overengineering: Now, who pays the price?

Last week Felipe and I were talking about some situations that we came across, really similar situations, although we never worked before on the same company, we can see situations on software development around us that has the origin overengineering, specially the overengineering on detailed design stuff(I know that “design” is a huge topic, so only few aspects we considered in this talk). We started to discuss about what motived in that time the team to design a so complex solution for products, and why they forgot accommodating changes that is one of the most challenging points of good software design. Most of those softwares that we were discussing about are medium and small projects and non mission-critical application, but have complexity on the maintenance cycle. Of course that quality of a design depends on which constraints are satisfied early and which are left for later, and most of the time some constraints doesn’t have a right/fast answer. And as managing complexity is something that catches my eye(don’t forget that when projects do fail for reasons that are technical, in most of the cases it is often uncontrolled complexity), we decided to make a brainstorm analyzing different points on design, as economic factors, layers, patterns, concurrence, test, quality attributes, how the abstract ideas were addressed and if they tried to make a component-based design and how they used the object-oriented design. After our discussion(considering the projects that we talked), we got the following points, that if maybe considered for those designers, could reduce the problems that we came across:

• On economic factors, we could see how designers have failed or been unable to anticipate future needs for change on an economic important issue that needs to be considered, that is the cost of making changes to a system. Specially the “Perfective maintenance”, that consist in extending and improving a system once it is operational, typically by providing new functionality/feature requested by users. For example, we could see designers “drawing” really complex UI solutions on top of MVC frameworks. But once the maintenance team receives new requirements that contains UI parts completely different about what was designed/thought, those new implementations become so hard and consume a lot time, where the price payed for a new “button” added on UI is unbelievable high.
• We thought about layers several times during the talk and it remembered how many times we’ve seen a layers mix to cause overengineering in the projects and how it introduced a significant complexity during the moment on which the development team needed to say about the kind of change will be coming. We were surprised about the bundle of extra layers that we identify in some softwares that has to be maintained, but always continue addition anything in terms of value for the software design itself.
• On patterns no question how they are important and lively for design process. I will assume that you are aware about architectural styles and design patterns. The enthusiasm of patterns adoption has sometime scarred me, taking a look in the design details is easy perceive that most of the time people are trying to find spots to push design patterns than using design pattern to solve a “common” problem that they came across. Other interesting point is the lack of care in creating documentation to express the reasons why those design patterns were adopted. Indeed, it can be argued that the enthusiasm for designing a better solution, these patterns has acted to expose the weaknesses in the “pattern concept adoption”. Then keep in mind the question of how useful it is to have an ever-increasing stock of patterns. The exaggerated mixed of patterns, arguing that it brings flexibility, probably it is not the right choice, because the complexity is coming together.
• About documentation the team responsible for maintain may find themselves with inadequate or inaccurate documentation, making their job hard to ensure that their changes are consistent with the original plans, and usually they are not, even becoming easy to introduce new bugs. And it is fact that the productivity is affected because when the team inherits an overengineering design, it is necessary to spend so much time learning the nuances of that design before the team can extend or maintain it, and probably the time constraint will be against you.
• During the design moment, abstraction performs an important role, where comes metaphors and representations. We can perceive that representations provide abstraction of a system and they are linked to the idea of views. Some designers have been disregarding the detailed design, in this way, not providing the different views for the design, not allowing to see the software throughout different points of view. A good example of views is “4+1 View”, or even using the alternative forms to observe the design, as constructional, behavioral, functional and data-modeling. Then basically it means that the concept of a view, as a means of capturing a particular set of design properties, and as projected through the use of a representation.
• Other interesting approach is keep the eyes on the construction planning, establishing the schedule to design(an of course debug and unit test), most of the time it tries to keep us “simpler”, specially when the project deviates from the schedule because we are spending more time on design that we “should”, and there are a lot chances of the reason is because the design is becoming hard/complex. The construction plan helps to organize the design(specially when project allocate more design activity to construction, usually on medium/small projects, allowing to flesh out details of the software design during the construction, and maybe it can naturally guides to a simpler design), define the order in which components are created and integrated and the software quality manage process, that directly weights on design quality. I know that is hard to know when the design is good enough, or how much details is enough or how much should be left to be done at the “typing moment”, and maybe these points helps on avoid  overengineering.
• On design for test, the suggestion is: keep in mind to design it to facilitate testing. It is really clear that when you design thinking on tests, result produced tends to be more formalized modules and classes, probably being more beneficial.

In additional, there is an interesting list of internal design characteristics that [McC04] mentions, and I like to have in mind:

• Minimal complexity
• Ease of maintenance
• Loose coupling
• Extensibility
• Reusability
• High fan-in
• Low-to-medium fan-out
• Portability
• Leannes
• Stratification
• Standard techniques

I remember an important passage from [PW85], where Parnas and Weiss, in their active design review approach, point interesting properties of a design as: it should be well structured, simple, efficient, adequate (satisfying the requirements), flexible (easy to change), practical (module interfaces sufficient for the job), implementable, and standardized (documentation organized in a standard way).

We know that design is a heuristic process, which means that no solution is guarantee to work for all projects and probably doesn’t fit on all teams, then the idea was just write down a small subset of things to consider during approaches to design and try to focus on reduced-complexity benefit, the easier and safer the software will be to change and test. We know that simplicity is just a vision, and “maybe” we can try to find it keeping these things in mind.

- [PW85] D. L. Parnas and D. M. Weiss. Active design reviews: Principles and practices. In Proceedings of the Eight International Conference on Software Engineering, pp. 215-222, 1985.
- [McC04] S. McConnell, Code Complete: A practical Handbook of Software Construction, Microsoft Press, second ed., 2004.