Implementing Linear Regression using Gradient Descent

In this blog I write about my experiences of implementing Linear regression using “Gradient Descent”.

In linear regression we try to model the relation between the features and the predictions as a linear relation. i,e Y = a + bX where a and b are called the coefficients and constant respectively. The problem of linear regression is to come up with values for these coefficients and the constant(parameters) such that the net error in predictions is least. The error is measured as :

The parameters of linear regression can be found out using the below code directly:

def get_linear_regression_parameters(X, y):
    X_with_constant = np.c_[np.ones((len(X),1)), X]
    a = np.matmul(X_with_constant.T, X_with_constant)
    b = np.matmul(X_with_constant.T, y)
    params = np.matmul(np.linalg.inv(a), b)
    return params

This implements the below expression:

(for proof refer to this)

We can also find the parameters using gradient descent to minimise the cost function. In python this can be done as shown below:

def gradient_descent(matrix_x, y, theta, learning_rate=0.01):
    length_of_y = len(y)
    theta_new = theta + 1
    while True:
        prediction = np.dot(matrix_x, theta)
        theta_new = theta - ((1/length_of_y) * learning_rate * (matrix_x.T.dot(prediction - y)))
        if np.linalg.norm(theta_new - theta) < 0.00001:
            return theta
        theta = theta_new
def linear_regression(data, target, learning_rate=0.01):
    theta = np.random.randn(14, 1)
    target = np.array(target,ndmin=2).T
    data_with_constant = np.c_[np.ones((len(data),1)), data]
    return gradient_descent(data_with_constant, target, theta)

The above code implements the logic as below

Implementing linear regression with gradient descent will allow us to verify the results from gradient descent using the direct implementation.