Are machines going to replace programmers?

Are machines going to replace programmers?

Or how software creates other software

I started doing some home baking recently. It started, like with a lot of other people, during the pandemic lockdown period when I got tired of buying the same bread from the supermarket every day. In all honesty, my bakes are passable, not very pretty but they please the family, which is good enough for me.

Yesterday I stumbled on a YouTube video on how a factory makes bread in synchronised perfection and it broke a bit of my heart. All the hard work kneading dough amounts to nothing compared to spinning motors tumbling through a mechanised giant bucket. As I watch rows and rows of dough rising in unison spirals up the proofing carousel then slowly rolling into a constantly humming monstrous oven to become marching loaves of bread, something died in me.

When the loaves zipped themselves into sealed bags and dumped themselves into packing boxes, I tell myself that they don’t have the same craftsmanship (in my mind) as someone who is making bread with love, for his family.

But deep inside me I understand that if bread depended on human bakers only, it would be a whole lot more expensive, a lot more people would go hungry. Automation and manufacturing lines have had always fed the masses since they were invented. The pride of a human baker matters little compared to feeding all the ever growing populations of human beings.

Fortunately, I don’t bake in my day job. I’m in the software development industry, where the main activity involves software programmers (developers, engineers, coders – take your pick) writing code to build simple to complex software systems that control much of the world today (even bread factories are controlled by manufacturing control systems). Programmers who are also humans. Programmers who are highly sought-after craftsman where their experience, talent and creativity is celebrated.

Just like bakers.

Of course this begs the question – are there software factories where software is churned out by machines, just like bread factories? Do they exist today?

Manufacturing line

Another perspective is that software development today is already a manufacturing line, with different software development teams working to take up different parts of the whole system to be assembled finally in well-organized processes.

In place of factory workers, we have programmers, in place of supervisors we have team leads and in place of a plant manager we have engineering managers. With a combination of outsourcing, offshoring, contracting as well as various manpower supplementary methods, companies can potentially ramp up software development as well.

Naturally no programmer likes to be called a factory worker. In fact, most of us prefer to called software craftsman . But there is no denying that somewhere along the way, developing software in a large-scale environment is very much like a factory.

Today, all large-scale software development happens in teams (doesn’t matter what it’s called or how much autonomy it’s given) and guided by some processes. Whether it’s agile, iterative, behaviour-driven, waterfall or any of the hybrid variants, most software development organizations have some sort of process and methodology. Larger organizations even have specialised teams to implement the processes and process models such as CMMI (Capability Maturity Model Integration) and ISO/IEC 12207 .

Some of these process models can get pretty serious. For example, CMMI which was developed by the Carnegie Mellon University (CMU) and the US Department of Defense, is required in many US government software development contracts .

While the idea of software development as a manufacturing line is not very appealing, and even counter to the image of the cool tech startup development, once software gets to a certain scale, structure and processes are part and parcel of production.

And as in baking bread, brewing beer and many other artisan jobs, making great software is a source of pride and joy. But when it comes to scale and production a process-oriented approach is critical to ensure quality, and more importantly, consistency.

However, as much as we build in processes, it’s all still done by human beings and is prone to the famous I D-10T error . That’s why over time, we have moved over to continuous automated integration, testing and deployment. The idea is to reduce as much human error as possible when integrating multiple parts and components of system we’re building by using scripts that automate the processes.

This works remarkably well but obviously there is still a gap — the system is designed by humans, the different parts and components are written by humans, even the automated scripts are written by humans. The system is only as robust as the weakest link and we are that link.

But what if every part of the process is automated?

Automation

In one of my favourite books, The Moon Is A Harsh Mistress (published 1966), Mike (short for Mycroft Holmes , Sherlock’s older brother) was an AI that “woke up”. Mike could understand classic programming but he could also understand Loglan (this is a real language) and English. Mannie (short for Manuel), the narrator and computerman (how cool is that name), in fact interacts with him mainly in English. Mike does his own programming based on their interactions. Imagine that. And this is in from mid-60s.

The idea that to program computers you only need to tell it your problems and it will create the necessary software to solve the programs is crazily compelling. In the 80s and 90s there were several movements to create 4th generation and 5th generation languages (4GL and 5GL) that will do exactly that, and they were marketed heavily.

The intention was that eventually actual coding wasn’t necessary, all it takes is to define the problem and the code will be generated to solve it. Naturally the focus was then to define the problem is a much more precise way so that better code can be generated. To do that, you’d need to build models and have a user-friendly way describing the problem such that even non software engineers can create software. My company then (like a lot of enterprises of its day) bought into it hook, line and sinker. To be honest, I fell for it too.

It never really took off because … programming languages were already doing that. To make a programming language suitable for non programmers you’d need to make it less rigid and more suitable for humans. But that would result in it being more fuzzy and less precise. In which case it can only create really simple, unambiguous software. To create more complex software you’d need someone who can not only define the problem but come up with the solution, write the code, make sure it runs and updates it as needed i.e. you need a programmer.

Not that 4GL/5GL wasn’t useful, just that its promises for the death of the software development industry was greatly exaggerated. The project I was working on crashed and burned, losing millions along the way (though I can’t blame it entirely on 4GL, it was partly the reason why it lost millions).

Software creating software

The 4GL/5GL wasn’t the answer partly because there’re still humans in the equation . What if we go deep and get software to create software? The developments in AI in the past few years, specifically machine learning, have resurfaced the idea. Machine learning is increasingly being used in lot of software that’s written today. There is already machine learning software today that are beating professional Go players, analysing markets, driving cars, and even creating music and art by learning from past masters. Is someone today creating some machine learning software that would write other software?

Let’s break down what that means. There are a few types of software that creates other software. First, there is self-reproduction, which really means how a piece of software can reproduce itself. The idea is not new, in fact, John von Neumann theorized about self-reproducing automata in the 1940s .

Quines

At the source code level, there are quines, which are programs that take no input and produces a copy of its own source code as its only output . Here’s a rather impressive quine called Qlobe created by Yusuke Endoh, a Ruby core member.

Run this in the command line:

curl -fSSl https://gist.githubusercontent.com/shime/f0ebe84ca42c33b51d42/raw/5e74315dc6b6fe572f8a457536ad7eb17ad3f1e4/qlobe.rb > qlobe.rb; while true; do ruby qlobe.rb | tee temp.rb; sleep 0.1; mv -f temp.rb qlobe.rb; done

You’ll see a spinning globe of code.

Worms

While quines are harmless fun, computer worms are not. Computer worms are standalone malware programs that self-reproduces and spreads across networks. The term “worm” was first used in the 1975 novel The Shockwave Rider by John Brunner . One of the first worms was the Morris worm , created by Robert Tappan Morris, a computer science graduate student from Cornell University in 1988. The Morris worm disrupted close to 10% of all computers on the Internet (not such a big number considering the size of the Internet then) in a virulent denial-of-service attack.

One of the most notorious worms is Stuxnet , which attacked industrial SCADA systems and is believe to have caused susbstantial damage to the Iran’s nuclear program. It is also believed to be a cyberweapon built jointly by US and Israel. I’ll let that sink in a while. State sponsored cyberattack on a nuclear facility.

Of course, self-reproduction is just a mechanism, it’s the payload that is malicious. Payloads can also be benign, though they are relatively rare. For example, the Welchia worm infects computers to combat the Blaster worm . It automatically downloads Microsoft security updates for Windows and reboots the computers, although it does that without any permissions and not everyone considers it friendly.

Genetic programming

What we talked about before is self-reproduction, which is basically software cloning a copy of itself. In the organic world, besides ameobas and other single cell animals, reproduction is more like producing offspring, creating newer generations of itself. This happens in software too!

Genetic programming (GP) is one way software can produce other software. GPs are population-based meta-heuristics that evolves programs that best solves a particular problem.

Alan Turing was the first to propose evolving programs in his 1950 paper, Computing Machinery and Intelligence . However, the GP and in general, the field of evolutionary algorithms was first laid out in John Holland’s classic book, Adaptation in Natural and Artificial Systems , first published in 1975.

GPs generally follow the principles of natural selection . These are the basic steps of a GP:

1. Generate the initial population of programs randomly (first generation)
2. Evaluate the fitness of each individual program
3. Select the best programs for reproduction of the next generation
4. Reproduce next generation programs from the previous generation of best solutions, through crossover and mutation mechanisms
5. Replace the less-fit programs with the next generation
6. Repeat steps 2 to 5 until the termination condition is met, usually when a suitable program to the problem is found

Programs are represented in memory as tree structures in GP, so traditionally GP favours the use of programming languages that naturally uses tree structures. I’ve talked about another type variant of this, called genetic algorithms (GA) in another story . Both GP and GA are variants of a field broadly known as evolutionary algorithms . While concepts for either are very similar, GP’s main representation are programs while GA’s main representation is in raw data (often in strings).

GPs are fascinating not only because it is something we as programmers did not code, but is evolved through generations. It’s also because it’s software that can continually change and adapt to the environment.

And it’s software not only creating software but literally giving birth to the next generation of software.

Deep neural networks

Deep neural networks (or deep learning) is a type of machine learning algorithm based on artificial neural networks. The word “deep” refers to the multiple layers in a neural network. Deep neural networks (DNN) has been getting a lot of attention in the recent years, with astonishing results and sometimes a bit of hype. I won’t delve too deeply (pun intended) into the how it works here but I’ve written a couple of stories on it, including one that shows how someone can build a neural network from ground up .

Neural networks aren’t new, the concepts date all the way to the 1950s, with many breakthroughs in the 1980s and 1990s. What really accelerated things were the breakthroughs and availability of computational power and growth of data. To use more buzzwords, cloud and big data made DNNs possible.

In 2014, DeepMind, a UK artificial intelligence company started a project called AlphaGo to build a deep neural network to compete in Go. By March 2016, it played Lee Sedol, one of the best Go players in the world and won 4 out of 5 games (using 1,920 CPUs and 280 GPUs). That 1 loss was the last time AlphaGo lost to a human being in an official game.

In May 2017, AlphaGo played 3 games with Ke Jie, the world No 1 ranked Go player and won all 3, after which it was awarded professional 9-dan by the Chinese Weiqi Association.

DNNs has been used in many many other fields, from image and speech recognition, to natural language processing and financial fraud detection. Self-driving vehicles (cars, buses, trucks) as well as autonomous robots (both ground-based and drones) depend heavily on DNNs. Cancer researchers use DNNs to detect cancer cells, DNNs has been used to generate music and art, even black and white photos have been recolored using DNNs.

But can DNNs be used to create software?

There are some attempts to generate source code but I don’t think that is really necessary. What we want is performed by the DNN models already — the DNN models are the software.

Let me explain what I mean.

The model is probably the software

In structured programming , we use sequence, selection (if-then-else) and iteration (for, while) to describe rules that processes data to produce results. Very often these rules are bundled into functions, which takes data input and produces results.

DNN models use multiple layers in multiple DNNs (many DNNs are assembled from different types of neural networks) to do the same. DNNs train models using massive amounts of data and once these models have been trained, they can be used to predict results, given new data. This means DNN models are functions that takes data input and products results too. In fact, DNN models can approximate and mimic any function!

The difference is that while structured code can be traced and followed, it is very difficult to trace the data flow through the different layers of a DNN model. This is because there are just too many parameters to be reasonably walked through. For example, LeNet-5 (published in 1998), one of the simplest convolutional neural network (CNN), has about 60,000 parameters . VGG-16 (published in 2014), a more recent CNN has about 138 million parameters!

In addition, the process to train models is stochastic and therefore the parameter values of the various nodes within the layers might be different even if we use the same training data!

Sure, after the training, DNNs are deterministic, meaning if you pass in the same data each time, the same results will be reproduced. However, many DNNs are designed to return a probabilistic result.

This might be confusing so let me give you an example. Let’s say we designed a DNN and train it with lots and lots of cat photos so that it can identify if a given photo has a cat.

Next we give it a picture. What will be DNN model return? It will be a probability of whether a cat is in the photo or not.

So how do we determine the accuracy of the model? We test it again with a whole bunch of photos we (as human beings) determine to be cat photos and we see how many it got correct or wrong. The correctness of the response is again determined by us. We can say if the probability is > 80% (or any number, really) it has a cat.

Let’s say we test it with 1,000 more cat photos and the model comes 90% of the time correct. What does that really mean? Well, we can decrease the correctness threshold and say that if probability result is > 70%, it means it is a cat, and then suddenly the model becomes 95% correct. Or if we train the model again with even more photos, and model becomes 97% correct. Or 85% correct, depending on the photos. Or 88% if we use a different set of 1,000 cat photos.

We don’t really know. The results are probabilistic, despite the model being deterministic.

How does it do it?

Wait, there’s a bit more than that. We know machine learning models like DNN models predict results. But actually we don’t really know how it does that. After training a model, we know it produces certain results, but we don’t know exactly why. It’s a black box.

But if it’s deterministic, we should know right?

Let’s start with a simple example. Let’s say we have a mathematical function f(x,y) where x = 1 and y = 2 that takes in 2 numbers, 1 and 2 and returns a third number 3 .

The function that immediately comes to mind is a simple addition, we take 1 and add 2 and we get 3 . But is that the only function? Actually no, we don’t really know what else happens in the function, so the if the function takes 1 and adds 100 then adds 2 and then subtracts 100 the function still returns 3 . For all purposes we can’t tell the difference. And there can be an infinite number of such functions, as long as we take in 1 and 2 and return 3 .

Now let’s say there is a mathematical function f(d) that can take in photos as an input and produces a probability whether there is a cat in it. Just as in our previous example, there could be any number of such functions that does this.

The DNN model we described above approximates f(d) by taking in the same input and producing the same output, but we don’t actually know which function it actually approximates. And we cannot say for sure it replicates any function either, unless we provide and test the finite set of data. But if we do that we don’t really need the DNN model anymore, do we?

So we end up with generated software which results and rules are no longer deterministic. This sounds totally useless except that DNN models can approximate any function and predict anything given suitable algorithms, lots of processing power and enough data.

This is its super power.

What does that all mean?

Now that we’ve done a bit of a tour, let’s get back to our original question — are machines going to replace programmers, will we lose our jobs?

So far as I can see nothing has taken any programmers’ job yet. If anything, it has intensified the need for more and better programmers. But what kind of programmers will be needed? That’s not a straightforward answer.

The current programming paradigm, which is structured and deterministic has been with us a long time, and will continue to be with us in the forseeable future. But much of the new exciting work will be in the field creating new machine learning algorithms, curating data pipelines, training and nurturing models.

New roles will come about to deal with the new probabilistic programming paradigm. For example, in software development, quality control activities are a critical part of the overall process. But how do you ensure a probabilistic function will always perform as needed? How do we ensure that we never miss a cat in a photo? Or say there’s a cat where there isn’t actually?

Genetic programs and DNN models are generated software. DNN models are generated without even source code and we don’t necessarily understand its rules and logic. That causes problems we don’t know how to resolve yet. For example, if an error comes about when the software runs, who is responsible? And how do we assure that it won’t happen again? This is not hypothetical. Tesla’s semi-autonomous Autopilot feature has been found to be partially responsible for fatal accidents .

As programmers, we need to navigate this new landscape and adapt to it. But as programmers we probably will. After all, our industry is all about constant change.