Over-engineering is a developer’s cry for help

Over-engineering is a developer’s cry for help – The Useless Dev blog

As always when I write about “anti patterns”, or “things not to do” – I’m speaking from experience.
In the case of over-engineering, not only did I use to do this “bad thing” – I still sometimes do it today – knowing full well it’s a “bad thing” to do!
For me, this proves a tough habit to break.

This is partly because I wasn’t actually clear, until recently, what “over-engineering” actually is.
I used to think that “over-engineering” is over-application of “engineering”. Like:

• Creating a 7-layer architecture for a CRUD app
• Using redux for a website with less than 5 pages
• Using kubernetes when you’re not google

But that’s not over-engineering. that’s just bad / overly-complex / resume-driven engineering.
Over-engineering, as I now understand it, is:

Building functionality that is not required.

(yes, “required” can be a bit blurry. For simplicity’s sake, we can define “required” as “appears on the work ticket we’re currently working on”)

On the face of it, this looks pretty simple. You’d have to be pretty dumb to work on something that nobody’s asked you to do. Right?

Well, what about “future proofing” and over-generalizing? Have you ever done those? I have do!
These are, actually, cases of building functionality that is not required.

As usual, XKCD explains it best:

How many times have you implemented a system to support any arbitrary condiment, when the customer just wanted some damn salt?

Some other examples of over-engineering, from personal experience:

• Recently I added a new string field to a data model. We just had to read it from a form, and later show it in the UI.
  A colleague suggested we write code to normalize the values in that field (e.g. downcase, trim whitespaces), as well as create an index in the DB for it. This is in case we’d need to filter or sort reports by that field in the future.
• I was tasked with building a very simple survey tool. All the questions asked would be “yes/no” questions.
  I implemented a system that can process any text answer.
• When calling out to a 3rd-party API, we wanted to retry 2 times when an error occurs, in case the error is transient.
  We’ve implemented a configurable system that retries X number of times.
• When a user submits an identifier of an OWASP top security risk (e.g. A03), we needed to show some information from OWASP about that risk.
  I was already planning a real-time API client or web-scraper, and how we can cache data even between requests to improve performance and resource utilization. That way we’ll always have up-to-date data in real time.
  My colleague just scraped the OWASP website into a JSON file, and set himself a reminder for next year to check the new top 10 list.

Also, have another look at the examples of “bad engineering” I gave above. In some contexts, they too can be considered over-engineering:
Building extra application layers “in case” we need to add more logic. Using kubernetes “in case” we need to handle web-scale load. etc.

Why is this a problem?

Some of the above “over-engineering” is not very complicated or difficult to do. Downcasing some strings, or reading from a configuration file. Is it really such a problem?

Keep in mind, though, that the cost of initial implementation is not the largest cost associated with writing code.

Any code that’s written requires maintenance.
Every developer looking at that code for the first time needs to figure out what it does, and why. How confusing is it when the answer to “why?” is “no reason”!
Every existing functionality needs to be preserved. So from now until forever, we need to be careful to not break it as we change the code around it. We need to regression-test it every time we make changes.

furthermore – when we guess at additional functionality, we guess that it may be required along some axis X.
And we write code based on some assumptions about X.

However, the actual functionality in the future may be along axis Y, which is different to X.
Now we’ve painted ourselves into a corner, designing our software to handle change in the wrong direction.

(For example – we designed a system to configure the number of retries for an API call.
But what if, in the future, we still want to only retry twice, BUT, we need to control the amount of time between retries? The “extra” code to check how many times to retry may make the implementation of configurable wait times more difficult)

Why do we do it?

Hopefully I’ve convinced you (or, you already knew) that implementing unwanted functionality is a bad idea.
So why would developers who are otherwise smart, skilled and reasonable, engage in over-engineering? Making their code harder to work with, in order to build functionality that nobody wants?

Are they crazy?

No. Quite the opposite – we over-engineer because we’re smart.

Here’s a very logical thought process where the logical conclusion is over-engineering:

1. It’s difficult to understand and / or to change this piece of code. AND / OR
2. It’s difficult / time consuming to validate that this piece of code is working as intended.
3. I currently have to change this piece of code, and validate that it works as intended (e.g. because I’m extending this functionality).
4. This is a difficult / time consuming process due to the above.
5. If I have to do this again in the future, this will result in more difficulty and time wasted.

Conclusion: As long as I’m here, I might as well make some further changes that may be needed in the future, even if they’re not needed now.
This will save the overhead of having to understand and / or validate this code again in the future.

Statistically, this seems like a sound conclusion.
Let’s say that I’m investing a further 1 hour of work on those not (yet) necessary features.
And let’s say that this 1 hour now can save me 3 hours in the future, if my guess about the future requirements is correct.
In that case, even if my guesses are only 34% correct, I still come out ahead.
I like dem odds!

However, if we look a bit closer, the above logic is flawed.
Specifically, the assumptions that we begin with:

The code is difficult to understand and / or change and / or validate.

But this is not due to some act or god, or a law of nature.
The code is difficult to understand because we wrote it this way.
It’s difficult to validate because we wrote it this way (or didn’t write good automated tests).

That’s why I claim that over-engineering is a cry for help:
If we start from the result (“we are over-engineering”) we can work our way back to the reason (“the code is difficult to work with”).

If we do that, we can understand the reason behind the need to over-engineer. Then we can hopefully deal with it, rather than doing the poor-person’s optimization of over-engineering.

Making our code safer and easier to change will make over-engineering unnecessary,
which will help to keep our code simpler,
which means it’ll be safer and easier to change,
which will make over-engineering unnecessary…
rinse and repeat.

On the other hand:
Over-engineering means that our code will be more complex (because we’re adding extra code and functionality),
which means it will be harder to understand and to change,
which will create an incentive to over-engineer,
which will make our code more complex…
rinse and repeat.

Which of these cycles would you rather be on?

I don’t want to ignore the fact that writing simple, easy-to-understand, well-tested code is hard. It’s a skill that i still haven’t mastered, 15 years in.

But if we don’t face this difficult challenge now, we’ll end facing the impossible challenge of changing an over-engineered mess.