Why is coding so ridiculously overcomplicated?

Don't get me wrong, it's not difficult... it's just way more fucking complex than it should be.

I'm coding some framework atm and decided to do it the right way... you know picking a language (and version...), choosing the right paradigm, doing TDD, creating CI/CD pipelines that trigger automatically when you raise a PR but can also be ran locally (oh and of course setting up github/azdo/bitbucket/whatever), choosing a license and prolly a whole lot more that I don't even realise. The only thing I didn't bother with (yet...) is code coverage. And all that before writing a single line of code.

And then totally unrelated I wrote the following script:

$ vim ~/bin/aur

#!/bin/bash

git pull

makepkg -i

And kept thinking how elegant and readable this is... and how I could royally fuck it up if I would do the right way.

Disclaimer

Just to be clear, this post has no point whatsoever. It merely raises a whole bunch of questions and maybe some solutions... or rather asks the reader (yeah that's you) to come up with solutions and hopefully share them.

Why do we do all this?

Honestly software engineering (aka coding) is an complete intertwined shit-show, which is so easy to dive into (Hello, World!) that everybody think they can do it... and consequently think they know better. So much so that people that never wrote a line of code are telling devs how to do their job. Which is probably the first hint at why we setup such an elaborate maze of tools, practices and processes...

Accountability

If something breaks, it's the dev's fault... not the PO for writing up a half-arsed feature or QA for not testing that 1 in a million scenario that he/she didn't have time for... or the CEO that doesn't want to spend more on their bread and butter.

Yeah that's right, software is your USP these days! The only way your gonna have an edge over your competition (in most businesses anyway) is to have the right tools and if those tools don't exist, you'll need to create them. You can have the best product/service in the world but if you don't have the software to back it up, your competitors will overtake you in an instance. That said, wouldn't it be nice if we could do it cheaper? Or at least smarter, cuz otherwise so could the competition...

Anyway, we're all accountable for the software we create, but we also need to be able to be accountable for it. So how can we be accountable for something ever-evolving and so complex that we can't even begin to understand what can possibly go wrong?

Over the years we came up with a whole bunch of things (ASD, CI/CD, DDD, FP, XP and the list goes on and on) in an attempt to solve the problems we faced. But what actually are those problems? And why do these solutions also create an equal amount of new problems? And why is there so much crap that I need to understand to write 1 line of code?

Fluidity

Probably the biggest problem that people generally tend to miss is the fluidity of software. When you build a (physical) house, it's pretty much fixed once it's done. You wouldn't decide the next day to add a swimming pool to the top-floor... and if you did, there would be at least 20 people telling you that, although awesome... it's a stupid fucking idea. In software however, the consequence of putting in a swimming pool isn't that it's gonna collapse and kill everyone in the building. It will weaken the foundation a bit but hey what's another bit of technical debt?

A fundamental part of this however is the ever-evolving nature of software. It's never finished and it never goes in the bin. Once a line of code is committed, it will likely stay there for the coming 10 years. And the problem with that is you didn't consider how that big-ass pool is gonna affect it, as no-one told you about the pool... or even considered it. As a matter of fact that 1 line of code may inspire someone to suggest that, if we can do this... we can also build a pool, right? What we create is often the inspiration for the next thing, potentially taking it in a completely new direction. But is the foundation strong enough for this?

And there is also the matter of all the legacy crap we keep carrying around. How do we ensure that that keeps working as expected? Cuz even though we wanna bolt new shiny things on top of it we can't loose those old (formerly shiny) things. Back in the day coding was a lot more like building a house, we'd do it once and a year later we'd build an improved version of it, having ample time to validate the old crap... not anymore.

Resilience

Although related to the fluidity in an architectural sense (how well does it cater for change?), resilience in software also relates to how well it runs. We expect software to always work but develop it in the narrowest possible conditions... "it works on my machine". And that's partly why a dev will always say it should work. It's so complex that dropping the exact same code on another machine typically will fail spectacularly!

So how is it that we created something so complex that we can't even predict whether it will work, let alone how well. Which is pretty impressive as everything we do is at the end of the day a very binary choice. But why than can't we deliver on the expectation that it always works, regardless of the conditions?

We need ensure all prerequisites are exactly met and otherwise it simply won't work. Tools like Docker will help us with that but there's still an endless amount of variables that we don't control... or even consider. Chaos engineering is one approach of making things more resilient but that's a requires a pretty hefty investment. Also it assume your software works, but what does that really mean? That it runs on your machine / prod cluster given the exact right circumstances? What if I want to run it on a new test cluster? How much shit would I need to setup before I even can start throwing a simian army at it?

Scale

If you write a quick little bash script like the AUR building example in the intro you can afford to ditch all the best practices. Or at least it's relatively safe to assume it's not gonna get extended or used anywhere else or by anyone else. But the bigger the stakes the more important such factors start to become. Lets say you want to provide a AUR building SAAS platform, that piece of code at the top is probably gonna be in there somewhere... but you might also wanna consider a microservices architecture... among 20000 other things.

It so depends on what you're end goal is on how "well" your code should be written. And the terrible truth is that half of the time we're figuring out the end goal along the way. So how can we make a proper judgement of how solid our code needs to be? Especially with a PO breathing down your neck who wants to get the feature out asap... and then bitches 2 weeks later that the next feature is taking too long cuz you need to clean up the mess of the first feature. So how can we write good enough code that don't have to rewrite/bin 2 weeks later? Or should we be writing throw-away code? And if so is there an architecture that actually facilitates this? And would the business even accept that? It is at the end of the day our IP, right?

Security

Another important (and royally underestimated) reason why such an overcomplicated mess of crap is needed, is security. We put locks on our doors to prevent unauthenticated and/or unauthorised people from coming in and additionally we also create secure briefcases to carry important information from A to B... and figure out how to deal with lock pickers and continuously monitor if our building is still up-to-date with (literally) today's standards. Or at least we should.

So how do you write secure code? How do you ensure your dependencies are secure (remember log4j?)? The long and short of it is: you can't... at best you can write software that takes more effort to break than the value gained if someone would break in. In other words the bigger the pot of gold your software is protecting the better security you need.

But half of the time people don't even know that the code they're writing is full of holes. And you can't really blame them as it's such a vast and complicated topic that fully understanding, in addition to all the other crap you need to know, is nearly impossible. So how do we create secure code then? Without having know every last little best practice or implement code scanners (and understanding and dealing with the results)... Or should we just deal with it in relation to the scale? i.e. initially the pot of gold is pretty small so who cares? But it would be so nice if it could be secure by default... like resilient by default...

Conclusion

So lets wrap this up for now. Normally you'd put some nice lessons learned here but there are none... other than coding being overly complex... but we kinda knew that when we started.

Here are a bunch of questions that I think we should consider... and that I might consider in future writings:

• How can we be fully accountable for what we build?

  • In other words how do we create control over what we build?

• How do we deal with the ever-evolving nature of software?

  • How do we stay in control over what we've built and what is still to come?

• Do we just start building and scale when needed?

  • Or can we easily build for scale?

• How can we build secure enough software?

  • Or can we make it 100% secure by default?

• Should we be proud of the vast amount of crap we created in the last 70+ years?

  • And how do we condense all this in 1 person's mind and expect them to create wonders?

Lastly, what if we had a greenfield? How would we deal with these problems? Could we do something radically different?