Understanding Linear Regression by Building It from Scratch with NumPy

Photo by Yana Vandeborne on Unsplash

Linear regression is one of the foundational algorithms in machine learning and statistics. Its simplicity and interpretability make it a popular choice for modeling relationships between variables. In this blog, we’ll dive into the core concepts of linear regression by building a model from scratch using Python and NumPy.

For more detailed implementations, check out the original repository:

• Linear Regression Code

GitHub - elcaiseri/Machine-Learning-from-Scratch: Machine Learning using NumPy

What is Linear Regression?

Linear regression aims to model the relationship between a dependent variable (target) and one or more independent variables (features) by fitting a linear equation to observed data. The equation of a simple linear regression model is:

[ y = wX + b ]

Where:

• ( y ) is the predicted output,
• ( X ) is the input feature(s),
• ( w ) represents the weights (coefficients),
• ( b ) is the bias (intercept).

Key Concepts

1. Weights and Biases

Weights and biases are the parameters the model learns during training. They determine the slope and intercept of the linear equation that best fits the data.

2. Loss Function

The loss function measures how well the model’s predictions match the actual data. For linear regression, the Mean Squared Error (MSE) is commonly used:

Mean Squared Error (MSE)

3. Gradient Descent

Gradient descent is an optimization algorithm used to minimize the loss function. It iteratively adjusts the weights and biases to reduce the error.

Implementing Linear Regression from Scratch

Step 1: Initialize Network Parameters

We start by defining a class for linear regression and initializing the parameters.

import numpy as np

class LinearRegression:
    def __init__(self, n_iter, learning_rate):
        self.n_iter = n_iter
        self.learning_rate = learning_rate
        self.w = None
        self.b = None

In the __init__ method, we initialize the number of iterations (n_iter) and the learning rate (learning_rate). These are hyperparameters that control the training process:

• n_iter: The number of iterations is the number of times the algorithm will run through the training data to update the weights. More iterations can lead to better learning, but also increase computational time.
• learning_rate: The learning rate determines the size of the steps the algorithm takes when adjusting the weights. A smaller learning rate can lead to more precise adjustments but may require more iterations, while a larger learning rate can speed up the training but risks overshooting the optimal weights.

We also initialize the weights (w) and bias (b) to None, which will be set during the training process.

Step 2: Fit Method

The fit method trains the model using gradient descent. It updates the weights and biases based on the gradient of the loss function.

def fit(self, X, y):
        n, m = X.shape
        self.w = np.zeros(m)
        self.b = 0

        for _ in range(self.n_iter):
            yhat = X.dot(self.w) + self.b
            error = yhat - y
            dw = (2 / n) * X.T.dot(error)
            db = (2 / n) * np.sum(error)
            self.w -= self.learning_rate * dw
            self.b -= self.learning_rate * db

Here, the fit method performs the following:

• Initialization: Sets the weights w to zeros with the same number of elements as there are features in X, and sets the bias b to zero.
• Gradient Descent Loop: For each iteration, it:
• Calculates the predicted values (yhat).
• Computes the error between the predicted and actual values.
• Calculates the gradients for the weights (dw) and bias (db).
• Updates the weights and bias using the gradients and the learning rate.

Step 3: Predict Method

The predict method uses the trained model to make predictions on new data.

def predict(self, X):
        return X.dot(self.w) + self.b

The predict method simply computes the dot product of the input features X with the weights w and adds the bias b to get the predicted values.

Example Usage

Here’s a complete example of how to use the LinearRegression class to fit a model and make predictions.

if __name__ == "__main__":
    from sklearn.model_selection import train_test_split as tts 
    from sklearn import datasets as ds 

    # Create dataset
    np.random.seed(42)    
    X, y = ds.make_regression(n_samples=10000, n_features=2, noise=5)
    X_train, X_test, y_train, y_test = tts(X, y, test_size=0.2)
    # Model
    lr = LinearRegression(n_iter=1000, learning_rate=0.01)
    lr.fit(X_train, y_train)
    y_pred = lr.predict(X_test)
    # Evaluate
    mse = np.mean((y_test - y_pred) ** 2)
    print("Linear Regression Test MSE:", mse)

In this example:

• We use scikit-learn to generate a synthetic dataset with 10,000 samples and 2 features.
• The dataset is split into training and testing sets.
• We create an instance of the LinearRegression class with 1000 iterations and a learning rate of 0.01.
• The model is trained on the training set.
• Predictions are made on the testing set.
• The mean squared error (MSE) is calculated to evaluate the model’s performance.

Conclusion

Building a linear regression model from scratch provides valuable insights into the underlying mechanics of machine learning algorithms. By implementing the core components yourself, you gain a deeper understanding of how models learn from data and make predictions. This foundational knowledge is essential for anyone looking to delve deeper into the world of machine learning.

Explore the code and experiment with different datasets to further solidify your understanding. Happy coding!