How a neuron in a 2D artificial neural network bends space in 3D — visualization

This is for the ones who want to get a visual understanding of in the inner workings of neurons in ANN. You just need a little bit of high-school level mathematics and a little patience. We will build a simple one that tells apples from oranges and then go deeper from there. Here goes.

Step 1: Go to a fruit market and buy apples and oranges.

Step 2: A nerd among us rates them on 2 attributes — red and sweet.

Voila — we have 2 dimensions now. X is for redness, and Y is for sweetness. This is what I call a 2D neuron i.e. one with 2 inputs.

Most of us would agree that apples are sweeter and redder in general, and hence we would end up with a graph looking something like the one below.

Let’s also draw a good-ole line to classify them. How we can come up with the specifications for a line is a topic for another story, but it doesn’t get in the way of our visualization.

2D graph for apples and oranges on a plane (x-axis: redness, y-axis: sweetness)

But what is a line anyway?

In slope-intercept format, it is y = mx+ c (where m is the slope and c is the y-intercept).

All good? With some simple rearrangement, this can be put as

w1 * x + w2 * y + b1 = 0

If I now take this equation and put some circles around it, it will start to look a whole like something we are so familiar with — yes this is our ANN fundamental building block AKA a neuron.

Now if we replace the 0 in the equation with a z , it will take our visualization to new dimension altogether — I mean literally.

w1 * x + w2 * y + b1 = z

OR z = w1 * x + w2 * y + b1

This (no prizes for guessing) is the equation of a plane in 3D. So any line on the X-Y axis in 2D is just a special case of the equation of a plane being set to 0 i.e. it is a set of all points where the plane intersects with the flat X-Y plane.

Neuron intermediate output without any activation

How about the activation?

The part I have deliberately left out so far is the activation on the neuron, so let’s put it in as well. I have used a ReLU, which just replaces all the negative value with zero. So, g(z) = max{0, z}

The pink sheet below is the final output from the neuron. If the sheet is flat on the floor, the fruit is an orange. If the sheet is up in the air, it’s an apple. Moreover, the farther up the sheet it, higher the probability it’s an apple.

Final neuron activation in 3D with a ReLU activation

Final neuron activation in 3D with a ReLU activation — different perspective

How does this output bending look like with multiple neurons?

I am working on the next page of this series to take the visualization to multiple neurons — will add a link once its published. Please leave me a comment if you think we should pick some other activation function (like a sigmoid) to spice it up. Thank you.