It’s popular to apply DNN (Deep neural network) in classification task. The famous MNIST dataset is one case. We can reshape the 784-dimensions input into 1-rank vector. Can we use DNN to solve regression problem?

The answer is yes. Remember we can just need to do like the classification work! The above code load the sine data which contain some noise. The result can be illustrated as the following:

Now we try to use regression model to fit these data distribution. First, we just use a simple DNN which just has one full-connected layer. The number of neuron is one, and you can regard it as a simple perceptron model.

However, the result is not pretty good. In our approach, we just multiply the input vector with weight tensor, and add with a bias tensor. You can regard as a linear model. Just like the following format:

output tensor = place_x * weight_tensor + bias_tensor

And the corresponding result is shown below:

The red line is the model we get. How to make it better? In the machine learning territory, it’s obvious that the linear model cannot solve the linear inseparable problem. One of the powerful technique we can use is to project the data to the high-dimension vector space! For example, we can project the whole data to the 3-dimension vector space. The following graph shows the project result:

The most left part is the original data. Through the full-connected layer, we can project the whole data to the 3-dimension space. The middle and the right part is the 3d model to show the project result.

In the above code, we can adjust the containing and make it become as a 2-layer DNN. It project the whole data to the 3-dimension and try to find the hyper-plane in the particular space. The following picture shows the result. You can see the model is more accurate than the previous one.

At final, a important property we should remember. The goal of machine learning isn’t to find the most perfect target function. On the contrary, we want to make the model update and like the target function as similar as possible.

The above images are all the possible hypothesis we can get. The left one is very normal. However, the middle one is a little not similar with the sine pattern. The right part is much awkward.