Understanding Linear Regression by Building It from Scratch with NumPy
Linear regression is one of the foundational algorithms in machine learning and statistics. Its simplicity and interpretability make it a popular choice for modeling relationships between variables. In this blog, we’ll dive into the core concepts of linear regression by building a model from scratch using Python and NumPy.
For more detailed implementations, check out the original repository:
What is Linear Regression?
Linear regression aims to model the relationship between a dependent variable (target) and one or more independent variables (features) by fitting a linear equation to observed data. The equation of a simple linear regression model is:
[ y = wX + b ]
Where:
- ( y ) is the predicted output,
- ( X ) is the input feature(s),
- ( w ) represents the weights (coefficients),
- ( b ) is the bias (intercept).
Key Concepts
1. Weights and Biases
Weights and biases are the parameters the model learns during training. They determine the slope and intercept of the linear equation that best fits the data.
2. Loss Function
The loss function measures how well the model’s predictions match the actual data. For linear regression, the Mean Squared Error (MSE) is commonly used:
3. Gradient Descent
Gradient descent is an optimization algorithm used to minimize the loss function. It iteratively adjusts the weights and biases to reduce the error.
Implementing Linear Regression from Scratch
Step 1: Initialize Network Parameters
We start by defining a class for linear regression and initializing the parameters.
import numpy as np
class LinearRegression:
def __init__(self, n_iter, learning_rate):
self.n_iter = n_iter
self.learning_rate = learning_rate
self.w = None
self.b = None
In the __init__
method, we initialize the number of iterations (n_iter
) and the learning rate (learning_rate
). These are hyperparameters that control the training process:
n_iter
: The number of iterations is the number of times the algorithm will run through the training data to update the weights. More iterations can lead to better learning, but also increase computational time.learning_rate
: The learning rate determines the size of the steps the algorithm takes when adjusting the weights. A smaller learning rate can lead to more precise adjustments but may require more iterations, while a larger learning rate can speed up the training but risks overshooting the optimal weights.
We also initialize the weights (w
) and bias (b
) to None
, which will be set during the training process.
Step 2: Fit Method
The fit
method trains the model using gradient descent. It updates the weights and biases based on the gradient of the loss function.
def fit(self, X, y):
n, m = X.shape
self.w = np.zeros(m)
self.b = 0
for _ in range(self.n_iter):
yhat = X.dot(self.w) + self.b
error = yhat - y
dw = (2 / n) * X.T.dot(error)
db = (2 / n) * np.sum(error)
self.w -= self.learning_rate * dw
self.b -= self.learning_rate * db
Here, the fit
method performs the following:
- Initialization: Sets the weights
w
to zeros with the same number of elements as there are features inX
, and sets the biasb
to zero. - Gradient Descent Loop: For each iteration, it:
- Calculates the predicted values (
yhat
). - Computes the error between the predicted and actual values.
- Calculates the gradients for the weights (
dw
) and bias (db
). - Updates the weights and bias using the gradients and the learning rate.
Step 3: Predict Method
The predict
method uses the trained model to make predictions on new data.
def predict(self, X):
return X.dot(self.w) + self.b
The predict
method simply computes the dot product of the input features X
with the weights w
and adds the bias b
to get the predicted values.
Example Usage
Here’s a complete example of how to use the LinearRegression
class to fit a model and make predictions.
if __name__ == "__main__":
from sklearn.model_selection import train_test_split as tts
from sklearn import datasets as ds
# Create dataset
np.random.seed(42)
X, y = ds.make_regression(n_samples=10000, n_features=2, noise=5)
X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2)
# Model
lr = LinearRegression(n_iter=1000, learning_rate=0.01)
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)
# Evaluate
mse = np.mean((y_test - y_pred) ** 2)
print("Linear Regression Test MSE:", mse)
In this example:
- We use
scikit-learn
to generate a synthetic dataset with 10,000 samples and 2 features. - The dataset is split into training and testing sets.
- We create an instance of the
LinearRegression
class with 1000 iterations and a learning rate of 0.01. - The model is trained on the training set.
- Predictions are made on the testing set.
- The mean squared error (MSE) is calculated to evaluate the model’s performance.
Conclusion
Building a linear regression model from scratch provides valuable insights into the underlying mechanics of machine learning algorithms. By implementing the core components yourself, you gain a deeper understanding of how models learn from data and make predictions. This foundational knowledge is essential for anyone looking to delve deeper into the world of machine learning.
Explore the code and experiment with different datasets to further solidify your understanding. Happy coding!