Gradient descent is an optimization algorithm used to find the set of parameters (coefficients) of a function that minimizes a cost function. This method iteratively adjusts the coefficients of the function until the cost reaches the local, or global, minimum. Gradient descent works by calculating the partial derivatives of the cost function with respect to each of the coefficients. The algorithm then makes small adjustments to the coefficients in order to reduce the cost until it reaches a minimum. The direction of the adjustments is determined by the negative of the gradient of the cost function. Optimizers are methods or algorithms that reduce a loss (an error) by adjusting various parameters and weights, minimizing the loss function, and thereby improving model accuracy and speed. One such optimization technique is RMSprop.

RMSProp (Root Mean Squared Propagation) is an adaptive learning rate optimization algorithm. It is an extension of the popular Adaptive Gradient Algorithm and is designed to dramatically reduce the amount of computational effort used in training neural networks. This algorithm works by exponentially decaying the learning rate every time the squared gradient is less than a certain threshold. This helps reduce the learning rate more quickly when the gradients become small. In this way, RMSProp is able to smoothly adjust the learning rate for each of the parameters in the network, providing a better performance than regular Gradient Descent alone.

The RMSprop algorithm utilizes exponentially weighted moving averages of squared gradients to update the parameters. Here is the mathematical equation for RMSprop:

1. Initialize parameters:
   • Learning rate: α
   • Exponential decay rate for averaging: γ
   • Small constant for numerical stability: ε
   • Initial parameter values: θ
2. Initialize accumulated gradients (Exponentially weighted average):
   • Accumulated squared gradient for each parameter: Et​= 0
3. Repeat until convergence or maximum iterations:
   • Compute the gradient of the objective function with respect to the parameters:

     *** QuickLaTeX cannot compile formula:
     g_t = \nabla_\theta J(\theta_t)
     
     
     *** Error message:
     Cannot connect to QuickLaTeX server: cURL error 28: Operation timed out after 40001 milliseconds with 0 bytes received
     Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
     These links might help in finding solution:
     http://wordpress.org/extend/plugins/core-control/
     http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

   • Update the exponentially weighted average of the squared gradients:
   • Update the parameters:

where,

• gt is the gradient of the loss function with respect to the parameters at time t
• is a decay factor
• Et​ is the exponentially weighted average of the squared gradients
• α is the learning rate
• ϵ is a small constant to prevent division by zero

This process is repeated for each parameter in the optimization problem, and it helps adjust the learning rate for each parameter based on the historical gradients. The exponential moving average allows the algorithm to give more importance to recent gradients and dampen the effect of older gradients, providing stability during optimization.

Implementation

Now, we will look into the implementation of the RMSprop. We will first import all the necessary libraries as follows.

Now, we will define our objective function and its derivatives. For this article we are considering the objective function to be

*** QuickLaTeX cannot compile formula:
5 \times x_1^2 + 7 \times x_2^2


*** Error message:
Cannot connect to QuickLaTeX server: cURL error 52: Empty reply from server
Please make sure your server/PHP settings allow HTTP requests to external resources ("allow_url_fopen", etc.)
These links might help in finding solution:
http://wordpress.org/extend/plugins/core-control/
http://wordpress.org/support/topic/an-unexpected-http-error-occurred-during-the-api-request-on-wordpress-3?replies=37

where x1 and x2 are variables.

Now, let us visualize this equation. We will look into its 3D graph between x1, x2, and y and we will also look into its 2D representation (contour plot).

Output:

Now, let us define our RMSprop optimizer.

Now, let us optimize our objective function using the RMSprop function.

Output:

The optimal value of x1 is: -0.10352260359924752
The optimal value of x2 is: 0.0025296212056016548
The optimal value of y is: 0.05362944016394148

Now, let us visualize the path or trajectory of the objective function.

Output: