Implementing Linear Regression using Gradient Descent
In this blog I write about my experiences of implementing Linear regression using “Gradient Descent”.
In linear regression we try to model the relation between the features and the predictions as a linear relation. i,e Y = a + bX where a and b are called the coefficients and constant respectively. The problem of linear regression is to come up with values for these coefficients and the constant(parameters) such that the net error in predictions is least. The error is measured as :
The parameters of linear regression can be found out using the below code directly:
def get_linear_regression_parameters(X, y):
X_with_constant = np.c_[np.ones((len(X),1)), X]
a = np.matmul(X_with_constant.T, X_with_constant)
b = np.matmul(X_with_constant.T, y)
params = np.matmul(np.linalg.inv(a), b)
return params
This implements the below expression:
We can also find the parameters using gradient descent to minimise the cost function. In python this can be done as shown below:
def gradient_descent(matrix_x, y, theta, learning_rate=0.01):
length_of_y = len(y)
theta_new = theta + 1
while True:
prediction = np.dot(matrix_x, theta)
theta_new = theta - ((1/length_of_y) * learning_rate * (matrix_x.T.dot(prediction - y)))
if np.linalg.norm(theta_new - theta) < 0.00001:
return theta
theta = theta_newdef linear_regression(data, target, learning_rate=0.01):
theta = np.random.randn(14, 1)
target = np.array(target,ndmin=2).T
data_with_constant = np.c_[np.ones((len(data),1)), data]
return gradient_descent(data_with_constant, target, theta)
The above code implements the logic as below
Implementing linear regression with gradient descent will allow us to verify the results from gradient descent using the direct implementation.