Regularization Techniques And Their Implementation In TensorFlow(Keras)

Understand conventional techniques used to mitigate the problem of overfitting within deep neural networks.

Regularization

Deep Neural Networks(DNN) have a vast amount of weights parameters internal to the architecture that learn a range of values. These range of values are the essential key to enabling the neural network to solve huge complex functions.

The deeper a neural network is, the more representational power it possesses, but, there is a shortcoming that occurs as the number of weight parameter increases. This shortcoming is that the neural network is more prone to overfitting the training dataset.

Overfitting:This problem involves the algorithm predicting new instances of patterns presented to it, based too closely on instances of patterns it observed and learnt during training. This can cause the machine-learning algorithm to not generalize accurately to unseen data. Overfitting can occur if the training data does not accurately represent the distribution of test data. Overfitting can be fixed by reducing the number of features in the training data and reducing the complexity of the network through various techniques

Regularization techniques reduce the possibility of a neural network overfitting by constraining the range of values that the weight values within the network hold( more on this later ).

This article introduces two regularization strategies that impose constraining terms on the results of loss functions.

The loss function is a method that quantifies ‘ how well ’ a machine learning model performs. The quantification is an output(cost) based on a set of inputs, which are referred to as parameter values. The parameter values are used to estimate a prediction, and the ‘loss’ is the difference between the predictions and the actual values.

This article won’t focus on the maths of regularization. Instead, this article presents some standard regularization methods and how to implement them within neural networks using TensorFlow(Keras).

For more details on the maths, these article by Raimi Karim and Renu Khandelwal present L1 and L2 regularization maths reasonably.