Welcome to your-notes.

∴

lorem ipsum dolor sit amet

— your-name

Welcome to Machine Learning notes.

∴

I completed the Machine Learning Specialization Course by taking detailed notes and summarizing critical concepts for future reference.

University of Stanford & DeepLearning.AI

Andrew Ng & Eddy Shyu

— emreaslan —

Supervised and Unsupervised Machine Learning

Introduction

  Machine learning is a branch of artificial intelligence that allows systems to learn and make predictions or decisions without explicit programming. Two main types of machine learning are Supervised Learning and Unsupervised Learning. Below is a summary of their characteristics, subfields, along with a visual representation for clarity.

graph TD
    A[Machine Learning] --> B[Supervised Learning]
    A --> C[Unsupervised Learning]
    B --> D[Regression]
    B --> E[Classification]
    C --> F[Clustering]
    C --> G[Association]
    C --> H[Dimensionality Reduction]

Supervised Learning

  Supervised learning is a type of machine learning where the model is trained on labeled data. Labeled data means that each input has a corresponding output (or target) already provided. The goal is for the model to learn the relationship between the inputs and outputs so that it can make predictions for new, unseen data.

Key Characteristics

• Input and Output: The training data contains both input features (X) and target labels (Y).
• Goal: Predict the output (Y) for a given input (X).

Subfields

1. Regression: Predicting continuous values (e.g., predicting rent prices based on apartment size).
2. Classification: Assigning inputs to discrete categories (e.g., diagnosing cancer as benign or malignant).

Example: Regression

• Scenario: Predicting rent prices based on apartment size (in m²).
• Details:
  • Input features (X): Apartment size, number of rooms, neighborhood, etc.
  • Target variable (Y): Rent price (e.g., $ per month).
• Model's Job: Learn the relationship between apartment features and rent prices, then predict the rent for a new apartment.

Example: Classification

• Scenario: Diagnosing cancer (e.g., benign or malignant tumor).
• Details:
  • Input features (X): Measurements like tumor size, texture, cell shape, etc.
  • Target variable (Y): Class label (e.g., "Benign" or "Malignant").
• Model's Job: Classify a new tumor as benign or malignant based on input features.

Unsupervised Learning

  Unsupervised learning deals with unlabeled data. The model tries to find patterns, structures, or relationships within the data without any predefined labels or targets. It’s often used for exploratory data analysis.

Key Characteristics

• Input Only: The data contains only input features (X), with no target labels (Y).
• Goal: Discover hidden patterns or groupings in the data.

Subfields

1. Clustering: Grouping similar data points into clusters (e.g., customer segmentation).
2. Dimensionality Reduction: Reducing the number of features in the dataset while preserving important information (e.g., PCA).
3. Association: Discovering relationships or associations between variables in large datasets (e.g., market basket analysis).

Example: Clustering

• Scenario: Grouping customers for targeted marketing.
• Details:
  • Input features (X): Customer age, income, purchase history, location, etc.
  • No predefined labels (Y).
• Model's Job: Identify clusters of customers (e.g., "High-spenders," "Budget-conscious buyers").

Example: Dimensionality Reduction

• Scenario: Visualizing high-dimensional data.
• Details:
  • Imagine you have a dataset with 100+ features (e.g., sensor data from a factory).
  • Dimensionality reduction (e.g., PCA) helps reduce it to 2D or 3D for easier visualization.
• Model's Job: Keep the important structure of the data while reducing complexity.

Example: Association

• Scenario: Market basket analysis to identify product associations.
• Details:
  • Input features (X): Transaction data showing items purchased together.
  • No predefined labels (Y).
• Model's Job: Identify rules like "If a customer buys bread, they are likely to buy butter."
• Use Case: Recommendation systems, inventory planning.

Comparison Table

Key Takeaways

• Supervised Learning requires labeled data and is commonly used for prediction tasks like regression and classification.
• Unsupervised Learning works with unlabeled data and focuses on finding hidden patterns through clustering or dimensionality reduction.
• Each technique has specific applications and is chosen based on the problem and the data available.

• Linear Regression and Cost Function
  • 1. Introduction
    • Why Linear Regression?
    • Real-World Example: Housing Prices
  • 2. Mathematical Representation
    • 2.1 Understanding the Linear Model
  • 3. Implementing Linear Regression Step by Step
    • 3.1 Import Necessary Libraries
    • 3.2 Generate Sample Data
    • 3.3 Visualizing the Data
    • 3.4 Plotting the Regression Line
      • 3.4.1 Compute the Slope (${\theta }_1$)
      • 3.4.2 Compute the Intercept (${\theta }_0$)
    • 3.5 Plotting the Regression Line
    • 3.6 Interpretation of the Regression Line
  • 4. Cost Function
    • 4.1 Example: Assuming ${\theta }_1=0$
      • Case 1: ${\theta }_0=1$
      • Case 2: ${\theta }_0=1.5$
      • Case 3: ${\theta }_0=2$ (Optimal Case)
      • Comparison

Linear Regression and Cost Function

1. Introduction

Linear regression is one of the fundamental algorithms in machine learning. It is widely used for predictive modeling, especially when the relationship between the input and output variables is assumed to be linear. The primary goal is to find the best-fitting line that minimizes the error between predicted values and actual values.

Why Linear Regression?

Linear regression is simple yet powerful for many real-world applications. Some common use cases include:

• Predicting house prices based on features like size, number of rooms, and location.
• Estimating salaries based on experience, education level, and industry.
• Understanding trends in various fields like finance, healthcare, and economics.

Real-World Example: Housing Prices

Consider predicting house prices based on the size of the house (in square meters). A simple linear relationship can be assumed: larger houses tend to have higher prices. This assumption is the foundation of our linear regression model.

2. Mathematical Representation

A simple linear regression model assumes a linear relationship between the input $x$ (house size in square meters) and the output $y$ (house price). It is represented as:

$$h_{\theta }(x)={\theta }_0+{\theta }_{1}x$$

where:

• $h_{\theta }(x)$ is the predicted house price.
• ${\theta }_0$ (intercept) and ${\theta }_1$ (slope) are the parameters of the model.
• $x$ is the house size.
• $y$ is the actual house price.

2.1 Understanding the Linear Model

But what does this equation really mean?

• ${\theta }_0$ (intercept): The price of a house when its size is 0 m².

• ${\theta }_1$ (slope): The increase in house price for every additional square meter.

For example, if:

• ${\theta }_0=50,000$ and ${\theta }_1=300$,

• A 100 m² house would cost: $h_{\theta }(100)=50000+300\cdot 100=80000$

• A 200 m² house would cost: $h_{\theta }(200)=50000+300\cdot 200=110000$

We can visualize this relationship using a regression line.

3. Implementing Linear Regression Step by Step

To make the theoretical concepts clearer, let's implement the regression model step by step using Python.

3.1 Import Necessary Libraries

import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

3.2 Generate Sample Data

np.random.seed(42)
x = 50 + 200 * np.random.rand(100, 1)  # House sizes in m² (50 to 250)
y = 50000 + 300 * x + np.random.randn(100, 1) * 5000  # House prices with noise

Here, we create a dataset with 100 samples, where:

• $x$ represents house sizes (random values between $50$ and $250$ m²).

• $y$ represents house prices, following a linear relation but with some noise.

3.3 Visualizing the Data

plt.figure(figsize=(8,6))
sns.scatterplot(x=x.flatten(), y=y.flatten(), color='blue', alpha=0.6)
plt.xlabel('House Size (m²)')
plt.ylabel('House Price ($)')
plt.title('House Prices vs Size')
plt.show()

3.4 Plotting the Regression Line

Before moving to cost function, let's fit a simple regression line to our data and visualize it.

In real-world applications, we don't manually compute these parameters. Instead, we use libraries like scikit-learn to perform linear regression efficiently.

3.4.1 Compute the Slope (${\theta }_1$)

theta_1 = np.sum((x - np.mean(x)) * (y - np.mean(y))) / np.sum((x - np.mean(x))**2)

Here, we compute the slope (${\theta }_1$) using the least squares method.

3.4.2 Compute the Intercept (${\theta }_0$)

theta_0 = np.mean(y) - theta_1 * np.mean(x)

This calculates the intercept (${\theta }_0$), ensuring that our regression line passes through the mean of the data.

3.5 Plotting the Regression Line

y_pred = theta_0 + theta_1 * x  # Compute predicted values

plt.figure(figsize=(8,6))
sns.scatterplot(x=x.flatten(), y=y.flatten(), color='blue', alpha=0.6, label='Actual Data')
plt.plot(x, y_pred, color='red', linewidth=2, label='Regression Line')
plt.xlabel('House Size (m²)')
plt.ylabel('House Price ($)')
plt.title('Linear Regression Model: House Prices vs. Size')
plt.legend()
plt.show()

3.6 Interpretation of the Regression Line

Now, what does this line tell us?

✅ If the slope ${\theta }_1$ is positive, then larger houses cost more (as expected).

✅ If the intercept ${\theta }_0$ is high, it means even the smallest houses have a significant base price.

✅ The steepness of the line shows how much price increases per square meter.

4. Cost Function

To measure how well our model is performing, we use the cost function. The most common cost function for linear regression is the Mean Squared Error (MSE):

$$J(\theta )=\frac{1}{2m}\sum (h\_\theta (x_i)-y_i{)}^2$$

where:

• $m$ is the number of training examples.
• $h\_\theta (x_i)$ is the predicted price for the $i-th$ house.
• $y_i$ is the actual price.

Any dashed line indicates an error. In the formula above, we calculated the sum of these, namely $J(\theta )$.

This function calculates the average squared difference between predicted and actual values, penalizing larger errors more. The goal is to minimize $J(\theta )$ to achieve the best model parameters.

4.1 Example: Assuming ${\theta }_1=0$

To illustrate how the cost function behaves, let's assume that ${\theta }_1=0$, meaning our model only depends on ${\theta }_0$. We'll use a small dataset with four x values and y values:

Since we assume ${\theta }_1=0$, our hypothesis function simplifies to: $$h_{\theta }(x)={\theta }_0\cdot x$$

We'll evaluate different values of ${\theta }_0$ and compute the corresponding cost function.

Case 1: ${\theta }_0=1$

For ${\theta }_0=1$, the predicted values are:

$$h_{\theta }(x)=1\cdot x=[1,2,3,4]$$

The error values:

$$\text{error}=h_{\theta }(x)-y=[1-2,2-4,3-6,4-8]=[-1,-2,-3,-4]$$

Computing the cost function:

$$J(\theta \ast 0=1)=\frac{1}{2m}\sum (h\ast \theta (x_i)-y_i{)}^2$$

$$J(1)=\frac{1}{8}((-1{)}^2+(-2{)}^2+(-3{)}^2+(-4{)}^2)=\frac{1}{8}(1+4+9+16)=\frac{30}{8}=3.75$$

Case 2: ${\theta }_0=1.5$

For ${\theta }_0=1.5$, the predicted values are:

$$h_{\theta }(x)=1.5\cdot x=[1.5,3,4.5,6]$$

The error values:

$$\text{error}=[1.5-2,3-4,4.5-6,6-8]=[-0.5,-1,-1.5,-2]$$

Computing the cost function:

$$J(1.5)=\frac{1}{8}((-0.5{)}^2+(-1{)}^2+(-1.5{)}^2+(-2{)}^2)$$

$$J(1.5)=\frac{1}{8}(0.25+1+2.25+4)=\frac{7.5}{8}=0.9375$$

Case 3: ${\theta }_0=2$ (Optimal Case)

For ${\theta }_0=2$, the predicted values match the actual values:

$$h_{\theta }(x)=2\cdot x=[2,4,6,8]$$

The error values:

$$\text{error}=[2-2,4-4,6-6,8-8]=[0,0,0,0]$$

Computing the cost function:

$$J(2)=\frac{1}{8}((0{)}^2+(0{)}^2+(0{)}^2+(0{)}^2)=0$$

Comparison

From our calculations:

• $J(1)=3.75$
• $J(1.5)=0.9375$
• $J(2)=0$

As expected, the cost function is minimized when ${\theta }_0=2$, which perfectly fits the dataset. Any deviation from this value results in a higher cost.

So how many times can the machine try and find the correct value? How can we teach it this? The answer is in the next topic.

• Introduction to Gradient Descent
• Mathematical Formulation of Gradient Descent
  • 1. Gradient Descent Update Rule
    • Why Do We Use the Derivative?
  • 2. Computing the Gradient
• Learning Rate ($\alpha$)
  • 1. When $\alpha$ is Too Small
  • 2. When $\alpha$ is Optimal
  • 3. When $\alpha$ is Too Large
  • Summary
• Gradient Descent Convergence
  • 1. Convergence Criteria
  • 2. Choosing the Right Stopping Condition
• Local Minimum vs Global Minimum
  • Understanding the Concept
  • Convex vs Non-Convex Cost Functions

Introduction to Gradient Descent

In the previous section, we explored how the cost function behaves when assuming different values of ${\theta }_0$ with ${\theta }_1=0$ (To visualize it easily, we give zero to ${\theta }_1$). Now, we introduce Gradient Descent, an optimization algorithm used to find the best parameters that minimize the cost function $J(\theta )$.

our hypothesis function simplifies to: $$h_{\theta }(x)={\theta }_0\cdot x$$

Gradient Descent is an iterative method that updates the parameter $\theta$ step by step in the direction that reduces the cost function. The algorithm helps us find the optimal value of ${\theta }_0$ efficiently instead of manually testing different values.

To understand how Gradient Descent works, let's recall our dataset:

We aim to find the best value of ${\theta }_0$ that minimizes the error between our predictions $h_{\theta }(x)={\theta }_0\cdot x$ and the actual $y$ values. Gradient Descent will iteratively adjust ${\theta }_0$ to reach the minimum cost.

Mathematical Formulation of Gradient Descent

Gradient Descent is an optimization algorithm used to minimize a function by iteratively updating its parameters in the direction of the steepest descent. In our case, we aim to minimize the cost function:

$$J(\theta )=\frac{1}{2m}\sum (h_{\theta }(x_i)-y_i{)}^2$$

Where:

• 𝑚 is the number of training examples.
• $h_{\theta }(x)$ represents our hypothesis function (predicted values).
• y represents the actual target values.
• Goal: Find the optimal $\theta$ that minimizes $J(\theta )$.

1. Gradient Descent Update Rule

Gradient Descent uses the derivative of the cost function to determine the direction and magnitude of updates. The general update rule for $\theta$ is:

$$\theta :=\theta -\alpha \frac{\partial J(\theta )}{\partial \theta }$$

Where:

• $\alpha$ (learning rate) controls the step size of updates.
• $\frac{\partial J(\theta )}{\partial \theta }$ is the gradient (derivative) of the cost function with respect to $\theta$.

Why Do We Use the Derivative?

The derivative $\frac{\partial J(\theta )}{\partial \theta }$ tells us the slope of the cost function. If the slope is positive, we need to decrease ${\theta }_0$ , and if it is negative, we need to increase ${\theta }_0$, guiding us toward the minimum of $J({\theta }_0)$ . Without derivatives, we wouldn't know which direction to move to minimize the function.

The gradient tells us how steeply the function increases or decreases at a given point.

• If the gradient is positive, $\theta$ is decreased.
• If the gradient is negative, $\theta$ is increased.

This ensures that we move toward the minimum of the cost function.

2. Computing the Gradient

First, recall our hypothesis function:

$$h_{\theta }(x)={\theta }_0\cdot x$$

Now, we compute the derivative of the cost function:

$$\frac{\partial J(\theta )}{\partial \theta \ast 0}=\frac{1}{m}\sum (h_{\theta }(x^{(i)})-y^{(i)})x^{(i)}$$

This expression represents the average gradient of the errors multiplied by the input values. Using this gradient, we update ${\theta }_0$ in each iteration:

$$\theta \ast 0:={\theta }_0-\alpha \cdot \frac{1}{m}\sum (h_{\theta }(x^{(i)})-y^{(i)})x^{(i)}$$

• If the error is large, the update step is bigger.
• If the error is small, the update step is smaller.

This way, the algorithm gradually moves towards the optimal ${\theta }_0$.

Learning Rate ($\alpha$)

The learning rate $(\alpha )$ is a crucial parameter in the gradient descent algorithm. It determines how large a step we take in the direction of the negative gradient during each iteration. Choosing an appropriate learning rate is essential for ensuring efficient convergence of the algorithm.

If the learning rate is too small, the algorithm will take tiny steps towards the minimum, leading to slow convergence. On the other hand, if the learning rate is too large, the algorithm may overshoot the minimum or even diverge, never reaching an optimal solution.

1. When $\alpha$ is Too Small

If the learning rate is set too small:

• Gradient descent will take very small steps in each iteration.
• Convergence to the minimum cost will be extremely slow.
• It may take a large number of iterations to reach a useful solution.
• The algorithm might get stuck in local variations of the cost function, slowing down learning.

Mathematically, the update rule is: ${\theta }_0:={\theta }_0-\alpha \frac{d}{d{\theta }_0}J({\theta }_0)$ When $\alpha$ is very small, the change in ${\theta }_0$ per step is minimal, making the process inefficient.

2. When $\alpha$ is Optimal

If the learning rate is chosen optimally:

• The gradient descent algorithm moves efficiently towards the minimum.
• It balances speed and stability, converging in a reasonable number of iterations.
• The cost function decreases steadily without oscillations or divergence.

A well-chosen $\alpha$ ensures that gradient descent follows a smooth and steady path to the minimum.

3. When $\alpha$ is Too Large

If the learning rate is set too large:

• Gradient descent may take excessively large steps.
• Instead of converging, it may oscillate around the minimum or diverge entirely.
• The cost function might increase instead of decreasing due to overshooting the optimal ${\theta }_0$.

In extreme cases, the cost function values might increase indefinitely, causing the algorithm to fail to find a minimum.

Summary

Selecting the right learning rate is essential for gradient descent to work efficiently. A well-balanced $\alpha$ ensures that the algorithm converges quickly and effectively. In the next section, we will implement gradient descent with different learning rates to visualize their effects.

Gradient Descent Convergence

Gradient Descent is an iterative optimization algorithm that minimizes the cost function, J(\theta), by updating parameters step by step. However, we need a proper stopping criterion to determine when the algorithm has converged.

1. Convergence Criteria

The algorithm should stop when one of the following conditions is met:

• Small Gradient: If the derivative (gradient) of the cost function is close to zero, meaning the algorithm is near the optimal point.
• Minimal Cost Change: If the difference in the cost function between iterations is below a predefined threshold ($|J(\theta \ast t)-J(\theta \ast t-1)|<\epsilon$).
• Maximum Iterations: A fixed number of iterations is reached to avoid infinite loops.

2. Choosing the Right Stopping Condition

• Stopping Too Early: If the algorithm stops before reaching the optimal solution, the model may not perform well.
• Stopping Too Late: Running too many iterations may waste computational resources without significant improvement.
• Optimal Stopping: The best condition is when further updates do not significantly change the cost function or parameters.

Local Minimum vs Global Minimum

Understanding the Concept

When optimizing a function, we aim to find the point where the function reaches its lowest value. This is crucial in machine learning because we want to minimize the cost function $J(\theta )$ effectively. However, there are two types of minima that gradient descent might encounter:

• Global Minimum: The absolute lowest point of the function. Ideally, gradient descent should converge here.
• Local Minimum: A point where the function has a lower value than nearby points but is not the absolute lowest value.

For convex functions (such as our quadratic cost function), gradient descent is guaranteed to reach the global minimum. However, for non-convex functions, the algorithm may get stuck in a local minimum.

Convex vs Non-Convex Cost Functions

1. Convex Functions

• The cost function $J(\theta )$ is convex for linear regression.
• This ensures that gradient descent always leads to the global minimum.
• Example: A simple quadratic function like $J(\theta )=(\theta -2{)}^2$.

2. Non-Convex Functions

• More common in deep learning and complex machine learning models.
• There can be multiple local minima.
• Example: Functions with multiple peaks and valleys, such as $J(\theta )=\sin (\theta )+\frac{{\theta }^2}{10}$.

Multiple Features

• Introduction
• Vectorization
  • Why Vectorization?
  • Vectorized Cost Function
• Feature Scaling
  • Why Feature Scaling is Important
  • Methods of Feature Scaling
    • 1. Min-Max Scaling (Normalization)
    • 2. Standardization (Z-Score Normalization)
  • Example
  • Feature Scaling in Gradient Descent

Introduction

In real-world scenarios, a single feature is often not enough to make accurate predictions. For example, if we want to predict the price of a house, using only its size (square meters) might not be sufficient. Other factors such as the number of bedrooms, location, and age of the house also play an important role.

When we have multiple features, our hypothesis function extends to:

$$h_{\theta }(x)={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_n{x}_n$$

where:

• $x_1,x_2,...,x_n$ are the input features,
• ${\theta }_0,{\theta }_1,...,{\theta }_n$ are the parameters (weights) we need to learn.

For instance, in a house price prediction model, the hypothesis function could be:

$$h_{\theta }(x)={\theta }_0+{\theta }_1(\text{Size})+{\theta }_2(\text{Number of Bedrooms})+{\theta }_3(\text{Age of House})$$

This allows our model to consider multiple factors, improving its accuracy compared to using a single feature.

Vectorization

To optimize computations, we represent our hypothesis function using matrix notation:

where:

$X$ is the matrix containing training examples

$\theta$ is the parameter vector

This allows efficient computation using matrix operations instead of looping over individual training examples.

Why Vectorization?

Vectorization is the process of converting operations that use loops into matrix operations. This improves computational efficiency, especially when working with large datasets. Instead of computing predictions one by one using a loop, we leverage linear algebra to perform all calculations simultaneously.

Without vectorization (using a loop):

m = len(X)  # Number of training examples
h = []
for i in range(m):
    prediction = theta_0 + theta_1 * X[i, 1] + theta_2 * X[i, 2] + ... + theta_n * X[i, n]
    h.append(prediction)

With vectorization:

h = np.dot(X, theta)  # Compute all predictions at once

This method is significantly faster because it takes advantage of optimized numerical libraries like NumPy that execute matrix operations efficiently.

Vectorized Cost Function

Similarly, our cost function for multiple features is:

$$J(\theta )=\frac{1}{2m}\sum (h_{\theta }(x^{(i)})-y^{(i)}{)}^2$$

Using matrices, this can be written as:

$$J(\theta )=\frac{1}{2m}(X\theta -y{)}^T(X\theta -y)$$

And implemented in Python as:

def compute_cost(X, y, theta):
    m = len(y)  # Number of training examples
    error = np.dot(X, theta) - y  # Compute (Xθ - y)
    cost = (1 / (2 * m)) * np.dot(error.T, error)  # Compute cost function
    return cost

By using vectorized operations, we achieve a significant performance boost compared to using explicit loops.

Feature Scaling

When working with multiple features, the range of values across different features can vary significantly. This can negatively affect the performance of gradient descent, causing slow convergence or inefficient updates. Feature scaling is a technique used to normalize or standardize features to bring them to a similar scale, improving the efficiency of gradient descent.

Why Feature Scaling is Important

• Features with large values can dominate the cost function, leading to inefficient updates.
• Gradient descent converges faster when features are on a similar scale.
• Helps prevent numerical instability when computing gradients.

Methods of Feature Scaling

1. Min-Max Scaling (Normalization)

Brings all feature values into a fixed range, typically between 0 and 1:

$$x_{scaled}^{(i)}=\frac{x^{(i)}-x_{min}}{x_{max}-x_{min}}$$

• Best for cases where the distribution of data is not Gaussian.
• Sensitive to outliers, as extreme values affect the range.

2. Standardization (Z-Score Normalization)

Centers data around zero with unit variance:

$$x_{scaled}^{(i)}=\frac{x^{(i)}-\mu }{\sigma }$$

where:

• $\mu$ is the mean of the feature values

• $\sigma$ is the standard deviation

• Works well when features follow a normal distribution.

• Less sensitive to outliers compared to min-max scaling.

Example

Consider a dataset with two features: House Size (m²) and Number of Bedrooms.

Using min-max scaling:

Feature Scaling in Gradient Descent

After scaling, gradient descent updates will be more balanced across different features, leading to faster and more stable convergence. Feature scaling is a critical preprocessing step in machine learning models involving optimization algorithms like gradient descent.

Feature Engineering and Polynomial Regression

• Feature Engineering
  • Introduction to Feature Engineering
    • Why is Feature Engineering Important?
    • Real-World Example
  • Feature Transformation
    • 1. Log Transformation
      • Example: Income Data
    • 2. Polynomial Features
      • Example: House Price Prediction
    • 3. Interaction Features
      • Example: Combining Features
  • Feature Selection
    • 1. Unnecessary Features
    • 2. Correlation Analysis
      • Example: Finding Highly Correlated Features
    • 3. Statistical Feature Selection Methods
      • Common Methods:
• Polynomial Regression
  • Introduction to Polynomial Regression
    • Why Use Polynomial Regression?
    • Example: Predicting House Prices
  • Mathematical Representation and Implementation

Feature Engineering

Introduction to Feature Engineering

Feature engineering is the process of transforming raw data into meaningful features that improve the predictive power of machine learning models. It involves creating new features, modifying existing ones, and selecting the most relevant features to enhance model performance.

Why is Feature Engineering Important?

• Improves model accuracy: Well-engineered features help models learn better representations of the data.
• Reduces model complexity: Properly engineered features can make complex models simpler and more interpretable.
• Enhances generalization: Good feature selection prevents overfitting and improves performance on unseen data.

Real-World Example

Consider a house price prediction problem. Instead of using just raw data such as square footage and the number of bedrooms, we can create new features like:

• Price per square foot = Price / Size
• Age of the house = Current Year - Year Built
• Proximity to city center = Distance in km

These engineered features often provide better insights and improve model performance compared to using raw data alone.

Feature Transformation

Feature transformation involves applying mathematical operations to existing features to make data more suitable for machine learning models.

1. Log Transformation

Used to reduce skewness and stabilize variance in highly skewed data.

Example: Income Data

Many income datasets have a right-skewed distribution where most values are low, but a few values are extremely high. Applying a log transformation makes the data more normal:

$$X^{\prime }=\log (X)$$

2. Polynomial Features

Adding polynomial terms (squared, cubic) to capture non-linear relationships.

Example: House Price Prediction

Instead of using Size as a single feature, we can include Size^2 and Size^3 to better fit non-linear patterns.

from sklearn.preprocessing import PolynomialFeatures
import numpy as np

X = np.array([[1000], [1500], [2000], [2500]])  # House sizes
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
print(X_poly)

3. Interaction Features

Creating new features based on interactions between existing ones.

Example: Combining Features

Instead of using Height and Weight separately for a health model, create a new BMI feature:

$$BMI=\frac{Weight}{Heigh{t}^2}$$

def calculate_bmi(height, weight):
    return weight / (height ** 2)

height = np.array([1.65, 1.75, 1.80])  # Heights in meters
weight = np.array([65, 80, 90])  # Weights in kg
bmi = calculate_bmi(height, weight)
print(bmi)

This allows the model to understand health risks better than using height and weight separately.

Feature Selection

Feature selection involves identifying the most relevant features for a model while removing unnecessary or redundant ones. This improves model performance and reduces computational complexity.

1. Unnecessary Features

Not all features contribute equally to model performance. Some may be irrelevant or redundant, leading to overfitting and increased computational cost. Examples of unnecessary features include:

• ID columns: Unique identifiers that do not provide predictive value.
• Highly correlated features: Features that contain similar information.
• Constant or near-constant features: Features with little to no variation.

2. Correlation Analysis

Correlation analysis helps detect multicollinearity, where two or more features are highly correlated. If two features provide similar information, one of them can be removed.

Example: Finding Highly Correlated Features

import pandas as pd
import numpy as np

# Sample dataset
data = {
    'Feature1': [1, 2, 3, 4, 5],
    'Feature2': [2, 4, 6, 8, 10],
    'Feature3': [5, 3, 6, 9, 2]
}
df = pd.DataFrame(data)

# Compute correlation matrix
correlation_matrix = df.corr()
print(correlation_matrix)

Features with a correlation coefficient close to ±1 can be considered redundant and removed.

3. Statistical Feature Selection Methods

Feature selection techniques can be used to rank the importance of different features based on statistical tests or model-based importance measures.

At this stage it is enough to learn superficially !

Common Methods:

• Chi-Square Test: Measures dependency between categorical features and the target variable.
• Mutual Information: Evaluates how much information a feature contributes.
• Recursive Feature Elimination (RFE): Iteratively removes less important features based on model performance.
• Feature Importance from Tree-Based Models: Decision trees and random forests provide feature importance scores.

Feature selection ensures that only the most valuable features are used in the final model, improving efficiency and predictive power.

Polynomial Regression

Introduction to Polynomial Regression

Polynomial Regression is an extension of Linear Regression that models non-linear relationships between input features and the target variable. While Linear Regression assumes a straight-line relationship, Polynomial Regression captures curves and more complex patterns.

Why Use Polynomial Regression?

• Handles Non-Linearity: Unlike Linear Regression, which assumes a direct relationship, Polynomial Regression models curved trends.
• Better Fit for Real-World Data: Many real-world phenomena, such as population growth, economic trends, and physics-based models, exhibit non-linear behavior.
• Feature Engineering Alternative: Instead of manually creating interaction terms, Polynomial Regression provides an automatic way to capture complex dependencies.

Example: Predicting House Prices

Consider a dataset where house prices do not increase linearly with size. Instead, they follow a non-linear trend due to factors like demand, location, and infrastructure. A Polynomial Regression model can better capture this pattern.

For instance:

• Linear Model: $Price={\beta }_0+{\beta }_1\cdot Size$
• Polynomial Model: $Price={\beta }_0+{\beta }_1\cdot Size+{\beta }_2\cdot Siz{e}^2$

This quadratic term helps model the curved price trend more accurately.

Mathematical Representation and Implementation

Polynomial regression extends linear regression by adding polynomial terms to the feature set. The hypothesis function is represented as:

$$h_{\theta }(x)={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+{\theta }_3{x}^3+...+{\theta }_n{x}^n$$

where:

• $x$ is the input feature,
• ${\theta }_0,{\theta }_1,...,{\theta }_n$ are the parameters (weights),
• $x^n$ represents higher-degree polynomial terms.

This allows the model to capture non-linear relationships in the data.

Classification with Logistic Regression

• 1. Introduction to Classification
  • Classification vs. Regression
  • Examples of Classification Problems
• 2. Logistic Regression
  • Introduction to Logistic Regression
  • Why Do We Need the Sigmoid Function?
  • Sigmoid Function and Probability Interpretation
  • Decision Boundary
    • Understanding the Decision Boundary with Examples
  • Key Takeaways
• 3. Cost Function for Logistic Regression
  • 1. Why Do We Need a Cost Function?
  • 2. Simplified Cost Function for Logistic Regression
  • 3. Intuition Behind the Cost Function
• 4. Gradient Descent for Logistic Regression
  • 1. Why Do We Need Gradient Descent?
  • 2. Gradient Descent Algorithm

1. Introduction to Classification

Classification is a supervised learning problem where the goal is to predict discrete categories instead of continuous values. Unlike regression, which predicts numerical values, classification assigns data points to labels or classes.

Classification vs. Regression

Examples of Classification Problems

• Email Spam Detection: Classify emails as "spam" or "not spam".
• Medical Diagnosis: Identify whether a patient has a disease (yes/no).
• Credit Card Fraud Detection: Determine if a transaction is fraudulent or legitimate.
• Image Recognition: Classifying images as "cat" or "dog".

Classification models can be:

• Binary Classification: Only two possible outcomes (e.g., spam or not spam).
• Multi-class Classification: More than two possible outcomes (e.g., classifying handwritten digits 0-9).

2. Logistic Regression

Introduction to Logistic Regression

Logistic regression is a statistical model used for binary classification problems. Unlike linear regression, which predicts continuous values, logistic regression predicts probabilities that map to discrete class labels.

Linear regression might seem like a reasonable approach for classification, but it has major limitations:

1. Unbounded Output: Linear regression produces outputs that can take any real value, meaning predictions could be negative or greater than 1, which makes no sense for probability-based classification.

2. Poor Decision Boundaries: If we use a linear function for classification, extreme values in the dataset can distort the decision boundary, leading to incorrect classifications.

To solve these issues, we use logistic regression, which applies the sigmoid function to transform outputs into a probability range between 0 and 1.

Why Do We Need the Sigmoid Function?

The sigmoid function is a key component of logistic regression. It ensures that outputs always remain between 0 and 1, making them interpretable as probabilities.

Consider a fraud detection system that predicts whether a transaction is fraudulent (1) or legitimate (0) based on customer behavior. Suppose we use a linear model:

$$y={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2$$

For some transactions, the output might be y = 7.5 or y = -3.2, which do not make sense as probability values. Instead, we use the sigmoid function to squash any real number into a valid probability range:

$$h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}}$$

This function maps:

• Large positive values to probabilities close to 1 (fraudulent transaction).
• Large negative values to probabilities close to 0 (legitimate transaction).
• Values near 0 to probabilities near 0.5 (uncertain classification).

Sigmoid Function and Probability Interpretation

The output of the sigmoid function can be interpreted as:

• $h_{\theta }(x)\approx 1$ → The model predicts Class 1 (e.g., spam email, fraudulent transaction).
• $h_{\theta }(x)\approx 0$ → The model predicts Class 0 (e.g., not spam email, legitimate transaction).

For a final classification decision, we apply a threshold (typically 0.5):

$$\hat{y}=\{\begin{array}{ll}1,& \text{if }{h}_{\theta }(x)\ge 0.5\\ 0,& \text{if }{h}_{\theta }(x)<0.5\end{array}$$

This means:

• If the probability is ≥ 0.5, we classify the input as 1 (positive class).
• If the probability is < 0.5, we classify it as 0 (negative class).

Decision Boundary

The decision boundary is the surface that separates different classes in logistic regression. It is the point at which the model predicts a probability of 0.5, meaning the model is equally uncertain about the classification.

Since logistic regression produces probabilities using the sigmoid function, we define the decision boundary mathematically as:

$$h_{\theta }(x)=\frac{1}{1+e^{-{\theta }^{T}x}}=0.5$$

Taking the inverse of the sigmoid function, we get:

$${\theta }^{T}x=0$$

This equation defines the decision boundary as a linear function in the feature space.

Understanding the Decision Boundary with Examples

1. Single Feature Case (1D)

If we have only one feature $x_1$, the model equation is:

$${\theta }_0+{\theta }_1{x}_1=0$$

Solving for $x_1$:

$$x_1=-\frac{{\theta }_0}{{\theta }_1}$$

This means that when $x_1$ crosses this threshold, the model switches from predicting Class 0 to Class 1.

Example: Imagine predicting whether a student passes or fails based on study hours ($x_1$):

• If $x_1<5$ hours → Fail (Class 0).
• If $x_1\ge 5$ hours → Pass (Class 1).

The decision boundary in this case is simply $x_1=5$.

2. Two Features Case (2D)

For two features $x_1$ and $x_2$, the decision boundary equation becomes:

$${\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2=0$$

Rearranging:

$$x_2=-\frac{{\theta }_0}{{\theta }_2}-\frac{{\theta }_1}{{\theta }_2}{x}_1$$

This represents a straight line separating the two classes in a 2D plane.

Example: Suppose we classify students as passing (1) or failing (0) based on study hours ($x_1$) and sleep hours ($x_2$):

• The decision boundary could be: $$x_2=-2-0.5x_1$$
• If $x_2$ is above the line, classify as pass.
• If $x_2$ is below the line, classify as fail.

3. Two Features Case (3D)

When we move to three features $x_1$, $x_2$, and $x_3$, the decision boundary becomes a plane in three-dimensional space:

$${\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3=0$$

Rearranging for $x_3$:

$$x_3=-\frac{{\theta }_0}{{\theta }_3}-\frac{{\theta }_1}{{\theta }_3}{x}_1-\frac{{\theta }_2}{{\theta }_3}{x}_2$$

This equation represents a flat plane dividing the 3D space into two regions, one for Class 1 and the other for Class 0.

Example:
Imagine predicting whether a company will be profitable (1) or not (0) based on:

• Marketing Budget ($x_1$)
• R&D Investment ($x_2$)
• Number of Employees ($x_3$)

The decision boundary would be a plane in 3D space, separating profitable and non-profitable companies.

In general, for n features, the decision boundary is a hyperplane in an n-dimensional space.

4. Non-Linear Decision Boundaries in Depth

So far, we have seen that logistic regression creates linear decision boundaries. However, many real-world problems have non-linear relationships. In such cases, a straight line (or plane) is not sufficient to separate classes.

To capture complex decision boundaries, we introduce polynomial features or feature transformations.

Example 1: Circular Decision Boundary

If the data requires a circular boundary, we can use quadratic terms:

$${\theta }_0+{\theta }_1{x}_1^2+{\theta }_2{x}_2^2=0$$

This represents a circle in 2D space.

For example:

• If $x_1$ and $x_2$ are the coordinates of points, a decision boundary like:

  $$x_1^2+x_2^2=4$$

  would classify points inside a radius-2 circle as Class 1 and outside as Class 0.

Example 2: Elliptical Decision Boundary

A more general quadratic equation:

$${\theta }_0+{\theta }_1{x}_1^2+{\theta }_2{x}_2^2+{\theta }_3{x}_1{x}_2=0$$

This allows for elliptical decision boundaries.

Example 3: Complex Non-Linear Boundaries

For even more complex boundaries, we can include higher-order polynomial features, such as:

$${\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_1^2+{\theta }_4{x}_2^2+{\theta }_5{x}_1{x}_2+{\theta }_6{x}_1^3+{\theta }_7{x}_2^3=0$$

This enables twists and curves in the decision boundary, allowing logistic regression to model highly non-linear patterns.

Feature Engineering for Non-Linear Boundaries

• Instead of adding polynomial terms manually, we can transform features using basis functions (e.g., Gaussian kernels or radial basis functions).
• Feature maps can convert non-linearly separable data into a higher-dimensional space where a linear decision boundary works.

Limitations of Logistic Regression for Non-Linear Boundaries

• Feature engineering is required: Unlike neural networks or decision trees, logistic regression cannot learn complex boundaries automatically.
• Higher-degree polynomials can lead to overfitting: Too many non-linear terms make the model sensitive to noise.

Key Takeaways

• In 3D, the decision boundary is a plane, and in higher dimensions, it becomes a hyperplane.
• Non-linear decision boundaries can be created using quadratic, cubic, or transformed features.
• Feature engineering is crucial to make logistic regression work well for non-linearly separable problems.
• Too many high-order polynomial terms can cause overfitting, so regularization is needed.

3. Cost Function for Logistic Regression

1. Why Do We Need a Cost Function?

In linear regression, we use the Mean Squared Error (MSE) as the cost function:

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i{)}^2$$

However, this cost function does not work well for logistic regression because:

• The hypothesis function in logistic regression is non-linear due to the sigmoid function.
• Using squared errors results in a non-convex function with multiple local minima, making optimization difficult.

We need a different cost function that:
✅ Works well with the sigmoid function.
✅ Is convex, so gradient descent can efficiently minimize it.

2. Simplified Cost Function for Logistic Regression

Instead of using squared errors, we use a log loss function:

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y_i\log (h_{\theta }(x_i))+(1-y_i)\log (1-h_{\theta }(x_i))\right]$$

Where:

• $y_i$ is the true label (0 or 1).
• $h_{\theta }(x_i)$ is the predicted probability from the sigmoid function.

This function ensures:

• If $y=1$ → The first term dominates: $-\log (h_{\theta }(x))$, which is close to 0 if $h\_\theta (x)\approx 1$ (correct prediction).
• If $y=0$ → The second term dominates: $-\log (1-h_{\theta }(x))$, which is close to 0 if $h\_\theta (x)\approx 0$.

✅ Interpretation: The function penalizes incorrect predictions heavily while rewarding correct predictions.

3. Intuition Behind the Cost Function

Let’s break it down:

• When $y=1$, the cost function simplifies to:

  $$-\log (h_{\theta }(x))$$

  This means:

  • If $h_{\theta }(x)\approx 1$ (correct prediction), $-\log (1)=0$ → No penalty.
  • If $h_{\theta }(x)\approx 0$ (wrong prediction), $-\log (0)\to \infty$ → High penalty!

• When $y=0$, the cost function simplifies to:

  $$-\log (1-h_{\theta }(x))$$

  This means:

  • If $h_{\theta }(x)\approx 0$ (correct prediction), $-\log (1)=0$ → No penalty.
  • If $h_{\theta }(x)\approx 1$ (wrong prediction), $-\log (0)\to \infty$ → High penalty!

✅ Key Takeaway:
The function assigns very high penalties for incorrect predictions, encouraging the model to learn correct classifications.

4. Gradient Descent for Logistic Regression

1. Why Do We Need Gradient Descent?

In logistic regression, our goal is to find the best parameters $\theta$ that minimize the cost function:

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y_i\log (h_{\theta }(x_i))+(1-y_i)\log (1-h_{\theta }(x_i))\right]$$

Since there is no closed-form solution like in linear regression, we use gradient descent to iteratively update $\theta$ until we reach the minimum cost.

2. Gradient Descent Algorithm

Gradient descent updates the parameters using the rule:

$${\theta }_j:={\theta }_j-\alpha \frac{\partial J(\theta )}{\partial {\theta }_j}$$

Where:

• $\alpha$ is the learning rate (step size).
• $\frac{\partial J(\theta )}{\partial {\theta }_j}$ is the gradient (direction of steepest increase).

For logistic regression, the derivative of the cost function is:

$$\frac{\partial J(\theta )}{\partial {\theta }_j}=\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i)x_{ij}$$

Thus, the update rule becomes:

$${\theta }_j:={\theta }_j-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i)x_{ij}$$

✅ Key Insight:

• We compute the error: $h_{\theta }(x_i)-y_i$.
• Multiply it by the feature $x\_ij$.
• Average over all training examples.
• Scale by $\alpha$ and update ${\theta }_j$.

Overfitting and Regularization

• 1. The Problem of Overfitting
  • What is Overfitting?
  • Symptoms of Overfitting
  • Example of Overfitting in Regression
    • Overfitting vs. Underfitting
    • Visualization of Overfitting
• 2. Addressing Overfitting
  • 1. Collecting More Data
  • 2. Feature Selection & Engineering
  • 3. Cross-Validation
  • 4. Regularization as a Solution
• 3. Regularized Cost Function
  • 1. Why Modify the Cost Function?
  • 2. Adding Regularization Term
    • L2 Regularization (Ridge Regression)
    • L1 Regularization (Lasso Regression)
  • 3. Effect of Regularization on Model Complexity
    • Visualization of Regularization Effects
• 4. Regularized Linear Regression
  • 1. Linear Regression Cost Function (Without Regularization)
  • 2. Regularized Cost Function for Linear Regression
  • 3. Effect of Regularization in Gradient Descent
    • Effect of Regularization on Parameters
  • 4. Normal Equation with Regularization
  • 5. Summary
• 5. Regularized Logistic Regression
  • 1. Logistic Regression Cost Function (Without Regularization)
  • 2. Regularized Cost Function for Logistic Regression
  • 3. Effect of Regularization in Gradient Descent
  • 4. Decision Boundary and Regularization
  • 5. Summary

1. The Problem of Overfitting

What is Overfitting?

Overfitting occurs when a machine learning model learns the training data too well, capturing noise and random fluctuations rather than the underlying pattern. As a result, the model performs well on training data but generalizes poorly to unseen data.

Symptoms of Overfitting

• High training accuracy but low test accuracy (poor generalization).
• Complex decision boundaries that fit training data too closely.
• Large model parameters (high magnitude weights), leading to excessive sensitivity to small changes in input data.

Example of Overfitting in Regression

Consider a polynomial regression model. If we fit a high-degree polynomial to data, the model may pass through all training points perfectly but fail to predict new data correctly.

Overfitting vs. Underfitting

Visualization of Overfitting

• Left (Underfitting): The model is too simple and cannot capture the trend.
• Middle (Good Fit): The model captures the pattern without overcomplicating.
• Right (Overfitting): The model follows the training data too closely, failing on new inputs.

2. Addressing Overfitting

Overfitting occurs when a model learns noise instead of the underlying pattern in the data. To address overfitting, we can apply several strategies to improve the model’s ability to generalize to unseen data.

1. Collecting More Data

• More training data helps the model capture real patterns rather than memorizing noise.
• Especially effective for deep learning models, where small datasets tend to overfit quickly.
• Not always feasible, but can be supplemented with data augmentation techniques.

2. Feature Selection & Engineering

• Removing irrelevant or redundant features reduces model complexity.
• Techniques like Principal Component Analysis (PCA) help reduce dimensionality.
• Engineering new features (e.g., creating polynomial features or interaction terms) can improve generalization.

3. Cross-Validation

• k-fold cross-validation ensures that the model performs well on different data splits.
• Helps detect overfitting early by testing the model on multiple subsets of data.
• Leave-one-out cross-validation (LOOCV) is another approach, especially useful for small datasets.

4. Regularization as a Solution

• Regularization techniques add constraints to the model to prevent excessive complexity.
• L1 (Lasso) and L2 (Ridge) Regularization introduce penalties for large coefficients.
• We will explore regularized cost functions in the next section.

By applying these techniques, we control model complexity and improve generalization performance. In the next section, we will dive deeper into regularization and its role in the cost function.

3. Regularized Cost Function

Overfitting often occurs when a model learns excessive complexity, leading to poor generalization. One way to control this is by modifying the cost function to penalize overly complex models.

1. Why Modify the Cost Function?

The standard cost function in regression or classification only minimizes the error on training data, which can result in large coefficients (weights) that overfit the data.

By adding a regularization term, we discourage large weights, making the model simpler and reducing overfitting.

2. Adding Regularization Term

Regularization adds a penalty term to the cost function that shrinks the model parameters. The two most common types of regularization are:

L2 Regularization (Ridge Regression)

In L2 regularization, we add the sum of squared weights to the cost function:

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m{\left[h_{\theta }(x^{(i)})-y^{(i)}\right]}^2+\lambda \sum _{j=1}^n{\theta }_j^2$$

• $\lambda$ (regularization parameter) controls how much regularization is applied.
• Higher $\lambda$ values force the model to reduce the magnitude of parameters, preventing overfitting.
• L2 regularization keeps all features but reduces their impact.

L1 Regularization (Lasso Regression)

In L1 regularization, we add the absolute values of weights:

$$J(\theta )=\frac{1}{m}\sum _{i=1}^m{\left[h_{\theta }(x^{(i)})-y^{(i)}\right]}^2+\lambda \sum _{j=1}^n|{\theta }_j|$$

• L1 regularization pushes some coefficients to zero, effectively performing feature selection.
• It results in sparser models, which are useful when many features are irrelevant.

3. Effect of Regularization on Model Complexity

Regularization controls model complexity by restricting parameter values:

• No Regularization ($\lambda =0$) → The model fits the training data too closely (overfitting).
• Small $\lambda$ → The model is still flexible but generalizes better.
• Large $\lambda$ → The model becomes too simple (underfitting), losing important patterns.

Visualization of Regularization Effects

• Left (No Regularization): The model overfits training data.
• Middle (Moderate Regularization): The model generalizes well.
• Right (Strong Regularization): The model underfits the data.

4. Regularized Linear Regression

Linear regression without regularization can suffer from overfitting, especially when the model has too many features or when training data is limited. Regularization helps by constraining the model's parameters, preventing extreme values that lead to high variance.

1. Linear Regression Cost Function (Without Regularization)

The standard cost function for linear regression is:

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2$$

where:

• $h\_\theta (x)={\theta }^{T}x$ is the hypothesis (predicted value),
• $m$ is the number of training examples.

This function minimizes the sum of squared errors but does not impose any restrictions on the parameter values, which can lead to overfitting.

2. Regularized Cost Function for Linear Regression

To prevent overfitting, we add an L2 regularization term (also known as Ridge Regression) to penalize large parameter values:

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m{\left(h_{\theta }(x^{(i)})-y^{(i)}\right)}^2+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$$

where:

• $\lambda$ is the regularization parameter that controls the penalty,
• The term $\sum {\theta }_j^2$ penalizes large values of $\theta$,
• ${\theta }_0$ (bias term) is not regularized.

3. Effect of Regularization in Gradient Descent

Regularization modifies the gradient descent update rule:

$${\theta }_j:={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j+\frac{\lambda }{m}{\theta }_j\right]$$

• The additional term $\frac{\lambda }{m}{\theta }_j$ shrinks the parameter values over time.
• When $\lambda$ is too large, the model underfits (too simple).
• When $\lambda$ is too small, the model overfits (too complex).

Effect of Regularization on Parameters

• If $\lambda =0$: Regularization is off → Overfitting risk.
• If $\lambda$ is too high: Model is too simple → Underfitting.
• If $\lambda$ is optimal: Good generalization → Balanced model.

4. Normal Equation with Regularization

For linear regression, we can solve for $\theta$ using the Normal Equation, which avoids gradient descent:

$$\theta =(X^{T}X+\lambda I{)}^{-1}{X}^{T}y$$

where:

• $I$ is the identity matrix (except ${\theta }_0$ is not regularized).
• Adding $\lambda I$ ensures $X^{T}X$ is invertible, reducing multicollinearity issues.

5. Summary

✅ Regularization reduces overfitting by penalizing large weights.
✅ L2 regularization (Ridge Regression) modifies cost function by adding $\sum {\theta }_j^2$.
✅ Gradient Descent and Normal Equation both adjust to include regularization.
✅ Choosing $\lambda$ is critical: too high → underfitting, too low → overfitting.

5. Regularized Logistic Regression

Logistic regression is commonly used for classification tasks, but like linear regression, it can overfit when there are too many features or limited data. Regularization helps control overfitting by penalizing large parameter values.

1. Logistic Regression Cost Function (Without Regularization)

The standard cost function for logistic regression is:

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\log h_{\theta }(x^{(i)})+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))\right]$$

where:

• $h\_\theta (x)=\frac{1}{1+e^{-{\theta }^{T}x}}$ is the sigmoid function,
• $y$ is the actual class label ($0$ or $1$),
• $m$ is the number of training examples.

This cost function does not include regularization, meaning the model may assign large weights to some features, leading to overfitting.

2. Regularized Cost Function for Logistic Regression

To reduce overfitting, we add an L2 regularization term, similar to regularized linear regression:

$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m\left[y^{(i)}\log h_{\theta }(x^{(i)})+(1-y^{(i)})\log (1-h_{\theta }(x^{(i)}))\right]+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$$

where:

• $\lambda$ is the regularization parameter (controls penalty),
• The term $\sum {\theta }_j^2$ discourages large parameter values,
• ${\theta }_0$ (bias term) is NOT regularized.

✅ Effect of Regularization

• Small $\lambda$ → Model may overfit (complex decision boundary).
• Large $\lambda$ → Model may underfit (too simple, missing important features).
• Optimal $\lambda$ → Model generalizes well.

3. Effect of Regularization in Gradient Descent

Regularization modifies the gradient descent update rule:

$${\theta }_j:={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m\left(h_{\theta }(x^{(i)})-y^{(i)}\right)x_j+\frac{\lambda }{m}{\theta }_j\right]$$

• The regularization term $\frac{\lambda }{m}{\theta }_j$ shrinks the weight values over time.
• Helps avoid models that memorize training data instead of learning patterns.

4. Decision Boundary and Regularization

Regularization also affects decision boundaries:

• Without regularization ($\lambda =0$): Complex boundaries that fit noise.
• With moderate $\lambda$: Simpler boundaries that generalize better.
• With very high $\lambda$: Too simplistic boundaries that underfit.

5. Summary

✅ Regularization in logistic regression prevents overfitting by controlling parameter sizes.
✅ L2 regularization (Ridge Regression) adds $\sum {\theta }_j^2$ to cost function.
✅ Gradient Descent is adjusted to shrink large weights.
✅ Choosing $\lambda$ is critical for a well-generalized model.

Scikit-learn: Practical Applications

• 1. Introduction to Scikit-Learn
  • Why Use Scikit-Learn?
  • Loading Built-in Datasets in Scikit-Learn
    • Example: Loading and Exploring the Iris Dataset
  • Splitting Data: Train-Test Split
    • Example: Splitting the Iris Dataset
  • Train-Test Split and Why It Matters
    • Why Not Use 100% of Data for Training?
    • Why Not Use 90% or More for Testing?
    • What’s the Ideal Train-Test Split?
• 2. Linear Regression with Scikit-Learn
  • 1. Introduction to Linear Regression
  • 2. Importing Required Libraries
  • 3. Creating a Sample Dataset
  • 4. Splitting Data into Training and Testing Sets
  • 5. Training the Linear Regression Model
  • 6. Making Predictions
  • 7. Evaluating the Model
  • 8. Visualizing the Results
• 3. Multiple Linear Regression with Scikit-Learn
  • What is Multiple Linear Regression?
  • Step 1: Generate a Synthetic Dataset
  • Step 2: Train the Model
  • Step 3: Visualize the Regression Plane
  • Key Takeaways
• 4. Polynomial Regression with Scikit-Learn
  • 1. What is Polynomial Regression?
  • 2. Generating Non-Linear Data
  • 3. Applying Polynomial Features
  • 4. Training a Polynomial Regression Model
  • 5. Visualizing the Results
  • 6. Comparing with Linear Regression
• 5. Binary Classification with Logistic Regression
  • 1. What is Logistic Regression?
  • 2. Generating a Synthetic Dataset (Spam Detection Example)
  • 3. Training the Logistic Regression Model
  • 4. Visualizing Decision Boundary
  • Key Takeaways
• 6. Multi-Class Classification with Logistic Regression
  • Step 1: Import Libraries
  • Step 2: Generate Synthetic Data
  • Step 3: Split the Data
  • Step 4: Train Logistic Regression Model
  • Step 5: Visualizing Decision Boundaries

1. Introduction to Scikit-Learn

Scikit-Learn is one of the most popular and powerful Python libraries for machine learning. It provides efficient implementations of various machine learning algorithms and tools for data preprocessing, model selection, and evaluation. It is built on top of NumPy, SciPy, and Matplotlib, making it highly compatible with the scientific computing ecosystem in Python.

Why Use Scikit-Learn?

• Easy to Use: Provides a simple and consistent API for machine learning models.
• Comprehensive: Includes a wide range of algorithms, including regression, classification, clustering, and dimensionality reduction.
• Efficient: Implements fast and optimized versions of ML algorithms.
• Integration: Works well with other libraries like Pandas, NumPy, and Matplotlib.

Loading Built-in Datasets in Scikit-Learn

Scikit-Learn provides several built-in datasets that can be used for practice and experimentation. Some common datasets include:

• Iris Dataset (load_iris): Classification dataset for flower species.
• Boston Housing Dataset (load_boston) (Deprecated): Regression dataset for predicting house prices.
• Digits Dataset (load_digits): Handwritten digit classification.
• Wine Dataset (load_wine): Classification dataset for different types of wine.
• Breast Cancer Dataset (load_breast_cancer): Binary classification dataset for cancer diagnosis.

Example: Loading and Exploring the Iris Dataset

from sklearn.datasets import load_iris
import pandas as pd

# Load the dataset
iris = load_iris()

# Convert to DataFrame
iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)

# Add target labels
iris_df['target'] = iris.target

# Display first few rows
print(iris_df.head())

Splitting Data: Train-Test Split

To evaluate a machine learning model, we need to split the data into a training set and a test set. This ensures that we can measure the model’s performance on unseen data.

Scikit-Learn provides train_test_split for this purpose:

Example: Splitting the Iris Dataset

from sklearn.model_selection import train_test_split

# Features and target variable
X = iris.data
y = iris.target

# Split into 80% training and 20% testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Training samples: {len(X_train)}, Testing samples: {len(X_test)}")

• test_size=0.2 means 20% of the data is reserved for testing.
• random_state=42 ensures reproducibility.

By following these steps, we have successfully loaded a dataset and prepared it for machine learning. In the next section, we will explore how to apply Linear Regression using Scikit-Learn.

Train-Test Split and Why It Matters

When training a machine learning model, we must evaluate its performance on unseen data to ensure it generalizes well. This is done by splitting the dataset into training and test sets.

Why Not Use 100% of Data for Training?

If we train the model using all available data, we won’t have any independent data to check how well it performs on new inputs. This leads to overfitting, where the model memorizes the training data instead of learning general patterns.

Why Not Use 90% or More for Testing?

While a large test set gives a better estimate of real-world performance, it reduces the amount of data available for training. A model trained on very little data may suffer from underfitting—it won’t have enough information to learn meaningful patterns.

What’s the Ideal Train-Test Split?

A commonly used ratio is 80% for training, 20% for testing. However, this depends on:

• Dataset Size: If data is limited, we may use a 90/10 split to keep more training data.
• Model Complexity: Simpler models may work with less training data, but deep learning models require more.
• Use Case: In critical applications (e.g., medical diagnosis), a larger test set (e.g., 30%) is preferred for reliable evaluation.

Key Takeaways

✅ 80/20 is a good starting point, but can vary based on dataset size and model needs.

✅ Too small a test set → Unreliable performance evaluation.

✅ Too large a test set → Model may not have enough training data to learn properly.

✅ Always shuffle the data before splitting to avoid biased results.

2. Linear Regression with Scikit-Learn

1. Introduction to Linear Regression

Linear regression is a fundamental supervised learning algorithm used to model the relationship between a dependent variable (target) and one or more independent variables (features). It assumes a linear relationship between input features and the output.

The mathematical form of a simple linear regression model is:

$$y={\theta }_0+{\theta }_{1}x$$

Where:

• $y$ is the predicted output.
• $x$ is the input feature.
• ${\theta }_0$ is the intercept (bias).
• ${\theta }_1$ is the coefficient (weight) of the feature.

Now, let's implement a simple linear regression model using Scikit-Learn.

2. Importing Required Libraries

First, we import necessary libraries for handling data, building the model, and evaluating its performance.

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

3. Creating a Sample Dataset

We will generate a synthetic dataset to train and test our linear regression model.

# Generate random data
np.random.seed(42)  # Ensures reproducibility
X = 2 * np.random.rand(100, 1)  # 100 samples, single feature
y = 4 + 3 * X + np.random.randn(100, 1)  # y = 4 + 3X + Gaussian noise

# Convert to a DataFrame for better visualization
df = pd.DataFrame(np.hstack((X, y)), columns=["Feature X", "Target y"])
df.head()

• np.random.rand(100, 1): Generates $100$ random values between $0$ and $2$.
• y = 4 + 3X + noise: Defines a linear relationship with some added noise.
• We use pd.DataFrame to display the first few samples.

4. Splitting Data into Training and Testing Sets

It is crucial to split the dataset into training and testing sets to evaluate model performance on unseen data.

# Splitting dataset into 80% training and 20% testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(f"Training set size: {X_train.shape[0]} samples")
print(f"Testing set size: {X_test.shape[0]} samples")

5. Training the Linear Regression Model

Now, we train a linear regression model using Scikit-Learn's LinearRegression() class.

# Create and train the model
model = LinearRegression()
model.fit(X_train, y_train)

# Print learned parameters
print(f"Intercept (theta_0): {model.intercept_[0]:.2f}")
print(f"Coefficient (theta_1): {model.coef_[0][0]:.2f}")

• fit(X_train, y_train): Trains the model by finding the best-fitting line.
• model.intercept_: The learned bias term.
• model.coef_: The learned weight for the feature.

6. Making Predictions

After training, we make predictions on the test set.

# Predict on test data
y_pred = model.predict(X_test)

# Compare actual vs predicted values
comparison_df = pd.DataFrame({"Actual": y_test.flatten(), "Predicted": y_pred.flatten()})
comparison_df.head()

• model.predict(X_test): Generates predictions.
• The DataFrame compares actual vs. predicted values.

7. Evaluating the Model

We use Mean Squared Error (MSE) and R² Score to evaluate model performance.

# Calculate Mean Squared Error (MSE)
mse = mean_squared_error(y_test, y_pred)

# Calculate R-squared score
r2 = r2_score(y_test, y_pred)

print(f"Mean Squared Error: {mse:.2f}")
print(f"R-squared Score: {r2:.2f}")

• MSE: Measures average squared differences between actual and predicted values (lower is better).
• R² Score: Measures how well the model explains the variance in the data (closer to 1 is better).

8. Visualizing the Results

Finally, let's plot the data and the regression line.

plt.scatter(X, y, color="blue", label="Actual Data")
plt.plot(X_test, y_pred, color="red", linewidth=2, label="Regression Line")
plt.xlabel("Feature X")
plt.ylabel("Target y")
plt.title("Linear Regression Model")
plt.legend()
plt.show()

This plot shows:

• Blue points → Actual test data
• Red line → Best-fit regression line

3. Multiple Linear Regression with Scikit-Learn

What is Multiple Linear Regression?

Multiple Linear Regression is an extension of simple linear regression where we predict a dependent variable ($y$) using multiple independent variables ($x_1,x_2,...,x_n$). The general form of the equation is:

$$y={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+...+{\theta }_n{x}_n$$

Where:

• $y$ = predicted output
• $x_1,x_2,...,x_n$ = independent variables (features)
• ${\theta }_0$ = intercept
• ${\theta }_1,{\theta }_2,...,{\theta }_n$ = coefficients (weights)

In this section, we will:

• Generate a synthetic dataset for a multiple linear regression model.
• Train a model using Scikit-Learn.
• Visualize the relationship in a 3D plot.

Step 1: Generate a Synthetic Dataset

First, let's create a dataset with two independent variables ($x_1$ and $x_2$) and one dependent variable ($y$). We'll add some noise to make it more realistic.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Set seed for reproducibility
np.random.seed(42)

# Generate random data for x1 and x2
x1 = np.random.uniform(0, 10, 100)
x2 = np.random.uniform(0, 10, 100)

# Define the true equation y = 3 + 2*x1 + 1.5*x2 + noise
y = 3 + 2*x1 + 1.5*x2 + np.random.normal(0, 2, 100)

# Reshape x1 and x2 for model training
X = np.column_stack((x1, x2))

Step 2: Train the Model

Now, we split the dataset into training and test sets and train a multiple linear regression model.

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train the model
model = LinearRegression()
model.fit(X_train, y_train)

# Get model parameters
theta0 = model.intercept_
theta1, theta2 = model.coef_
print(f"Model equation: y = {theta0:.2f} + {theta1:.2f}*x1 + {theta2:.2f}*x2")

Step 3: Visualize the Regression Plane

Since we have two independent variables ($x_1$ and $x_2$), we can plot the regression plane in 3D space.

# Generate grid for x1 and x2
x1_range = np.linspace(0, 10, 20)
x2_range = np.linspace(0, 10, 20)
x1_grid, x2_grid = np.meshgrid(x1_range, x2_range)

# Compute predicted y values
y_pred_grid = theta0 + theta1 * x1_grid + theta2 * x2_grid

# Create 3D plot
fig = plt.figure(figsize=(10, 6))
ax = fig.add_subplot(111, projection='3d')

# Scatter plot of real data
ax.scatter(x1, x2, y, color='red', label='Actual data')

# Regression plane
ax.plot_surface(x1_grid, x2_grid, y_pred_grid, alpha=0.5, color='cyan')

# Labels
ax.set_xlabel('X1')
ax.set_ylabel('X2')
ax.set_zlabel('Y')
ax.set_title('Multiple Linear Regression: 3D Visualization')
plt.legend()
plt.show()

Key Takeaways

• We generated a dataset with two independent variables and one dependent variable.
• We trained a Multiple Linear Regression model using Scikit-Learn.
• We visualized the regression plane in 3D, showing how $x_1$ and $x_2$ influence $y$.

4. Polynomial Regression with Scikit-Learn

Polynomial Regression is an extension of Linear Regression, where we introduce polynomial terms to capture non-linear relationships in the data.

1. What is Polynomial Regression?

Linear regression models relationships using a straight line:

$$y={\theta }_0+{\theta }_{1}x$$

However, if the data follows a non-linear pattern, a straight line won't fit well. Instead, we can introduce polynomial terms:

$$y={\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+{\theta }_3{x}^3+\cdots +{\theta }_n{x}^n$$

This allows the model to capture curvature in the data.

2. Generating Non-Linear Data

First, let's create a synthetic dataset with a non-linear relationship.

import numpy as np
import matplotlib.pyplot as plt

# Generate random x values between -3 and 3
np.random.seed(42)
X = np.linspace(-3, 3, 100).reshape(-1, 1)

# Generate a non-linear function with some noise
y = 0.5 * X**3 - X**2 + 2 + np.random.randn(100, 1) * 2

# Scatter plot of the data
plt.scatter(X, y, color='blue', alpha=0.5, label="True Data")
plt.xlabel("X")
plt.ylabel("y")
plt.title("Generated Non-Linear Data")
plt.legend()
plt.show()

• We create 100 random points between -3 and 3.
• The function we generate follows a cubic equation:
• $y=0.5x^3-x^2+2$ with added noise.
• We visualize the data using a scatter plot.

3. Applying Polynomial Features

To transform our linear features into polynomial features, we use PolynomialFeatures from sklearn.preprocessing.

from sklearn.preprocessing import PolynomialFeatures

# Transform X into polynomial features (degree=3)
poly = PolynomialFeatures(degree=3)
X_poly = poly.fit_transform(X)

print(f"Original X shape: {X.shape}")
print(f"Transformed X shape: {X_poly.shape}")
print(f"First 5 rows of X_poly:\n{X_poly[:5]}")

• We use PolynomialFeatures(degree=3) to add polynomial terms up to $x^3$.
• This converts each $x$ value into a feature vector $[1,x,x^2,x^3]$.
• We print the new shape and first few transformed rows.

4. Training a Polynomial Regression Model

Now, we train a Linear Regression model using these polynomial features.

from sklearn.linear_model import LinearRegression

# Train polynomial regression model
model = LinearRegression()
model.fit(X_poly, y)

# Predictions
y_pred = model.predict(X_poly)

5. Visualizing the Results

Let's plot the polynomial regression model against the actual data.

plt.scatter(X, y, color='blue', alpha=0.5, label="True Data")
plt.plot(X, y_pred, color='red', linewidth=2, label="Polynomial Regression Fit")
plt.xlabel("X")
plt.ylabel("y")
plt.title("Polynomial Regression Model")
plt.legend()
plt.show()

6. Comparing with Linear Regression

Now, let's compare Polynomial Regression with a simple Linear Regression model.

# Train a simple Linear Regression model
linear_model = LinearRegression()
linear_model.fit(X, y)
y_linear_pred = linear_model.predict(X)

# Plot both models
plt.scatter(X, y, color='blue', alpha=0.5, label="True Data")
plt.plot(X, y_pred, color='red', linewidth=2, label="Polynomial Regression Fit")
plt.plot(X, y_linear_pred, color='green', linestyle="dashed", linewidth=2, label="Linear Regression Fit")
plt.xlabel("X")
plt.ylabel("y")
plt.title("Polynomial vs. Linear Regression")
plt.legend()
plt.show()

5. Binary Classification with Logistic Regression

Logistic Regression is a fundamental algorithm used for binary classification problems. It estimates the probability that a given input belongs to a particular class using the sigmoid function.

1. What is Logistic Regression?

Unlike Linear Regression, which predicts continuous values, Logistic Regression predicts probabilities and then maps them to class labels (0 or 1). The model is defined as:

$$P(y=1|X)=\frac{1}{1+e^{-{\theta }^{T}X}}$$

Where:

• $\theta$ represents the model parameters (weights and bias).
• $X$ represents the input features.
• The output is a probability between 0 and 1.

2. Generating a Synthetic Dataset (Spam Detection Example)

We'll create a synthetic dataset where emails are classified as spam (1) or not spam (0) based on two features:

1. Number of suspicious words
2. Email length

import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Generating synthetic data
np.random.seed(42)
num_samples = 200

# Feature 1: Number of suspicious words (randomly chosen values)
suspicious_words = np.random.randint(0, 20, num_samples)

# Feature 2: Email length (short emails tend to be spammy)
email_length = np.random.randint(20, 300, num_samples)

# Labels: Spam (1) or Not Spam (0)
labels = (suspicious_words + email_length / 50 > 10).astype(int)

# Creating feature matrix
X = np.column_stack((suspicious_words, email_length))
y = labels

# Splitting into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

3. Training the Logistic Regression Model

Now, we train a Logistic Regression model on our dataset.

# Training the model
model = LogisticRegression()
model.fit(X_train, y_train)

# Making predictions
y_pred = model.predict(X_test)

# Evaluating the model
accuracy = accuracy_score(y_test, y_pred)
print(f"Model Accuracy: {accuracy:.2f}")

4. Visualizing Decision Boundary

The decision boundary helps us see how the model separates spam from non-spam emails. We plot the boundary in 2D.

# Function to plot decision boundary
def plot_decision_boundary(model, X, y):
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 10, X[:, 1].max() + 10
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 100),
                         np.linspace(y_min, y_max, 100))

    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)

    plt.contourf(xx, yy, Z, alpha=0.3, cmap=plt.cm.coolwarm)
    plt.scatter(X[:, 0], X[:, 1], c=y, edgecolors='k', cmap=plt.cm.coolwarm)
    plt.xlabel("Suspicious Words Count")
    plt.ylabel("Email Length")
    plt.title("Logistic Regression Decision Boundary")
    plt.show()

# Plotting the decision boundary
plot_decision_boundary(model, X, y)

This plot shows how the model separates spam and non-spam emails using our two features.

Key Takeaways

• Logistic Regression is used for binary classification.
• It estimates probabilities using the sigmoid function.
• We generated a synthetic dataset mimicking spam detection.
• We trained and evaluated a Logistic Regression model.
• Decision boundaries help visualize how the model classifies data.

6. Multi-Class Classification with Logistic Regression

In this section, we will implement a Multi-Class Classification model using Logistic Regression. Instead of a binary classification problem, we will classify data points into three distinct categories.

This project predicts a student's success level based on study hours and past grades using Logistic Regression.

We classify students into three categories:

• Fail (0)
• Pass (1)
• High Pass (2)

Step 1: Import Libraries

We start by importing necessary libraries for:

• Data generation
• Visualization
• Model training

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, classification_report

Step 2: Generate Synthetic Data

We create artificial student data using make_classification.

Each student has:

• Past Grades (0-100)
• Study Hours (non-negative)

We set random_state = 457897 to ensure reproducibility.

# Generate a classification dataset
X, y = make_classification(n_samples=300,
                           n_features=2,
                           n_classes=3,
                           n_clusters_per_class=1,
                           n_informative=2,
                           n_redundant=0,
                           random_state=457897)  # Ensures consistent results

# Normalize Study Hours to be non-negative & scale Past Grades (0-100)
X[:, 0] = X[:, 0] * 12
X[:, 1] = X[:, 1] * 100

# Scatter plot of generated data
plt.figure(figsize=(7, 5))
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='viridis', edgecolors='k', alpha=0.75)
plt.xlabel("Study Hours")
plt.ylabel("Past Grades")
plt.title("Student Performance Dataset")
plt.colorbar(label="Class (0: Fail, 1: Pass, 2: High Pass)")
plt.show()

Step 3: Split the Data

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=457897, stratify=y)

# Standardizing features for better model performance
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Step 4: Train Logistic Regression Model

from sklearn.multiclass import OneVsRestClassifier

# Define and train the model
model = OneVsRestClassifier(LogisticRegression(solver='lbfgs'))
model.fit(X_train, y_train)

Step 5: Visualizing Decision Boundaries

# Define a mesh grid for visualization
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 5, X[:, 1].max() + 5
xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                     np.linspace(y_min, y_max, 200))

# Predict on the mesh grid
Z = model.predict(scaler.transform(np.c_[xx.ravel(), yy.ravel()]))
Z = Z.reshape(xx.shape)

# Plot decision boundary
plt.figure(figsize=(7, 5))
plt.contourf(xx, yy, Z, alpha=0.3, cmap="viridis")
plt.scatter(X[:, 0], X[:, 1], c=y, cmap="viridis", edgecolors='k', alpha=0.75)
plt.xlabel("Study Hours")
plt.ylabel("Past Grades")
plt.title("Decision Boundaries of Student Performance Classification")
plt.colorbar(label="Class (0: Fail, 1: Pass, 2: High Pass)")
plt.show()

Neural Networks: Intuition and Model

• Understanding Neural Networks
• Biological Inspiration: The Brain and Synapses
  • Structure of a Biological Neuron
  • Artificial Neural Networks vs. Biological Networks
• Importance of Layers in Neural Networks
  • Example: Predicting a T-shirt's Top-Seller Status
• Face Recognition Example: Layer-by-Layer Processing
• Mathematical Representation of a Neural Network
  • Example Calculation
  • Two Hidden Layer Neural Network Calculation
    • Network Structure
    • First Hidden Layer Calculation
    • Second Hidden Layer Calculation
    • Output Layer Calculation
  • Conclusion
  • Handwritten Digit Recognition Using Two Layers
    • First Layer: Feature Extraction
    • Second Layer: Pattern Recognition
    • Output Layer: Classification

Understanding Neural Networks

Neural networks are a fundamental concept in deep learning, inspired by the way the human brain processes information. They consist of layers of artificial neurons that transform input data into meaningful outputs. At the core of a neural network is a simple mathematical operation: each neuron receives inputs, applies a weighted sum, adds a bias term, and passes the result through an activation function. This process allows the network to learn patterns and make predictions.

Biological Inspiration: The Brain and Synapses

Artificial neural networks (ANNs) are designed based on the biological structure of the human brain. The brain consists of billions of neurons, interconnected through structures called synapses. Neurons communicate with each other by transmitting electrical and chemical signals, which play a critical role in learning, memory, and decision-making processes.

Structure of a Biological Neuron

Each biological neuron consists of several key components:

• Dendrites: Receive input signals from other neurons.
• Cell Body (Soma): Processes the received signals and determines whether the neuron should be activated.
• Axon: Transmits the output signal to other neurons.
• Synapses: Junctions between neurons where chemical neurotransmitters facilitate communication.

Artificial Neural Networks vs. Biological Networks

In artificial neural networks:

• Neurons function as computational units.
• Weights correspond to synaptic strengths, determining how influential an input is.
• Bias terms help shift the activation threshold.
• Activation functions mimic the way biological neurons fire only when certain thresholds are exceeded.

Importance of Layers in Neural Networks

Neural networks are composed of multiple layers, each responsible for extracting and processing features from input data. The more layers a network has, the deeper it becomes, allowing it to learn complex hierarchical patterns.

Example: Predicting a T-shirt's Top-Seller Status

Consider an online clothing store that wants to predict whether a new T-shirt will become a top-seller. Several factors influence this outcome, which serve as inputs to our neural network:

• Price ($x_1$)
• Shipping Cost ($x_2$)
• Marketing ($x_3$)
• Material ($x_4$)

These inputs are fed into the first layer of the network, which extracts meaningful features. A possible hidden layer structure could be:

1. Hidden Layer 1: Contains a few activations functions like: affordability , awareness, perceived quality.
2. Output Layer: Aggregates information from the previous layers to make a final prediction.

The output layer applies a sigmoid activation function:

$$\sigma (z)=\frac{1}{1+e^{-z}}$$

where $z$ is a weighted sum of the previous layer’s outputs. If $\sigma (z)>0.5$, we classify the T-shirt as a top-seller; otherwise, it is not.

Face Recognition Example: Layer-by-Layer Processing

Face recognition is a real-world example where neural networks excel. Let's consider a deep neural network designed for face recognition, breaking down the processing step by step:

1. Input Layer: An image of a face is converted into pixel values (e.g., a 100x100 grayscale image would be represented as a vector of 10,000 pixel values).

2. First Hidden Layer: Detects basic edges and corners in the image by applying simple filters.
3. Second Hidden Layer: Identifies facial features like eyes, noses, and mouths by combining edge and corner information.
4. Third Hidden Layer: Recognizes entire facial structures and relationships between features.

5. Output Layer: Determines whether the face matches a known identity by producing a probability score.

Mathematical Representation of a Neural Network

To efficiently compute activations in a neural network, we use matrix notation. The general formula for forward propagation is:

$$Z^{[l]}=W^{[l]}{A}^{[l-1]}+b^{[l]}$$

where:

• $A^{[l-1]}$ is the activation from the previous layer,
• $W^{[l]}$ is the weight matrix of the current layer,
• $b^{[l]}$ is the bias vector,
• $Z^{[l]}$ is the linear combination of inputs before applying the activation function.

The activation function is applied as:

$$A^{[l]}=g(Z^{[l]})$$

where $g$ is typically a sigmoid, ReLU, or softmax function.

Example Calculation

Suppose we have a single-layer neural network with three inputs and one neuron. We define the inputs as:

$$x_1=0.5,x_2=0.8,x_3=0.2$$

The corresponding weight matrix and bias term are given by:

$$W=\left[\begin{array}{ccc}0.9& -0.5& 0.3\end{array}\right],b=0.1$$

The weighted sum (Z) is calculated as:

$$Z=W\cdot X+b=(0.5\times 0.9)+(0.8\times -0.5)+(0.2\times 0.3)+0.1$$

$$Z=0.45-0.4+0.06+0.1=0.21$$

Applying the sigmoid activation function:

$$\sigma (Z)=\frac{1}{1+e^{-Z}}=\frac{1}{1+e^{-0.21}}\approx 0.552$$

Since the output is above 0.5, we classify this case as positive.

Two Hidden Layer Neural Network Calculation

Now, let's consider a neural network with two hidden layers.

Network Structure

• Input Layer: 3 input values $X=[x_1,x_2,x_3]$
• First Hidden Layer: 4 neurons
• Second Hidden Layer: 3 neurons
• Output Layer: 1 neuron

First Hidden Layer Calculation

Given input vector:

$$X=\left[\begin{array}{c}0.5\\ 0.8\\ 0.2\end{array}\right]$$

Weight matrix for the first hidden layer:

$$W^{(1)}=\left[\begin{array}{ccc}0.2& -0.3& 0.5\\ -0.7& 0.1& 0.4\\ 0.3& 0.8& -0.6\\ 0.5& -0.2& 0.7\end{array}\right]$$

Bias vector:

$$b^{(1)}=\left[\begin{array}{c}0.1\\ -0.2\\ 0.3\\ 0.4\end{array}\right]$$

Computing the weighted sum:

$$Z^{(1)}=W^{(1)}X+b^{(1)}$$

Applying the sigmoid activation function:

$$A^{(1)}=\sigma (Z^{(1)})$$

Second Hidden Layer Calculation

Weight matrix:

$$W^{(2)}=\left[\begin{array}{cccc}0.6& -0.1& 0.3& 0.7\\ 0.2& 0.9& -0.5& 0.4\\ -0.3& 0.5& 0.7& -0.6\end{array}\right]$$

Bias vector:

$$b^{(2)}=\left[\begin{array}{c}-0.1\\ 0.3\\ 0.2\end{array}\right]$$

Computing the weighted sum:

$$Z^{(2)}=W^{(2)}{A}^{(1)}+b^{(2)}$$

Applying the sigmoid activation function:

$$A^{(2)}=\sigma (Z^{(2)})$$

Output Layer Calculation

Weight matrix:

$$W^{(3)}=\left[\begin{array}{ccc}0.5& -0.7& 0.6\end{array}\right]$$

Bias:

$$b^{(3)}=-0.2$$

Computing the final weighted sum:

$$Z^{(3)}=W^{(3)}{A}^{(2)}+b^{(3)}$$

Applying the sigmoid activation function:

$$A^{(3)}=\sigma (Z^{(3)})$$

If $A^{(3)}>0.5$, the output is classified as positive.

Conclusion

1. The first hidden layer extracts basic features.
2. The second hidden layer learns more abstract representations.
3. The output layer makes the final classification decision.

This demonstrates how a multi-layer neural network processes information in a hierarchical manner.

Handwritten Digit Recognition Using Two Layers

A classic application of neural networks is handwritten digit recognition. Let's consider recognizing the digit '1' from an 8x8 pixel grid using a simple neural network with two layers.

First Layer: Feature Extraction

• The 8x8 image is flattened into a 64-dimensional input vector.
• This vector is processed by neurons in the first hidden layer.
• The neurons identify edges, curves, and simple shapes using learned weights.
• Mathematically, the output of the first layer can be represented as:

$$Z^{(1)}=W^{(1)}X+b^{(1)}$$ $$A^{(1)}=\sigma (Z^{(1)})$$

Second Layer: Pattern Recognition

• The first layer's output is passed to a second hidden layer.
• This layer detects digit-specific features, such as the vertical stroke characteristic of '1'.
• The transformation at this stage follows:

$$Z^{(2)}=W^{(2)}{A}^{(1)}+b^{(2)}$$ $$A^{(2)}=\sigma (Z^{(2)})$$

Output Layer: Classification

• The final layer has 10 neurons, each representing a digit from 0 to 9.
• The neuron with the highest activation determines the predicted digit:

$$Z^{(3)}=W^{(3)}{A}^{(2)}+b^{(3)}$$ $$\text{Prediction}=\arg \max (A^{(3)})$$

This structured approach demonstrates how neural networks model real-world problems, from binary classification to deep learning applications like face and handwriting recognition.

Implementation of Forward Propagation

• Coffee Roasting Example (Classification Task)
• Neural Network Architecture
• TensorFlow Implementation
  • Step 1: Importing Libraries
  • Step 2: Defining Inputs and Outputs
  • Step 3: Building the Model
  • Step 4: Training the Model
  • Step 5: Making Predictions
• Forward Propagation Step-by-Step (NumPy Implementation)
  • Initializing Weights and Biases
  • Forward Propagation Calculation
• Artificial General Intelligence (AGI)
  • Everyday Example: AGI vs. Narrow AI
  • Key Challenges in AGI
  • Is AGI Possible?

Coffee Roasting Example (Classification Task)

Imagine we want to classify coffee as either "Good" or "Bad" based on two factors:

• Temperature (°C)
• Roasting Time (minutes)

For simplicity, we define:

• Good coffee: If the temperature is between 190°C and 210°C and the roasting time is between 10 and 15 minutes.
• Bad coffee: Any other condition.

We collect the following data:

We will implement a simple neural network using TensorFlow to classify new coffee samples.

Neural Network Architecture

We construct a neural network using the following structure:

• Input Layer: Two neurons (temperature, time)
• Hidden Layer: Three neurons, activated with the sigmoid function
• Output Layer: One neuron, activated with the sigmoid function (binary classification)

TensorFlow Implementation

Step 1: Importing Libraries

import tensorflow as tf
import numpy as np

• tensorflow is the core deep learning library that allows us to define and train neural networks.
• numpy is used for handling arrays and numerical operations efficiently.

Step 2: Defining Inputs and Outputs

X = np.array([[200, 12], [180, 10], [210, 15], [220, 20], [195, 13]], dtype=np.float32)
y = np.array([[1], [0], [1], [0], [1]], dtype=np.float32)

• X represents the input features (temperature and roasting time) as a NumPy array.
• y represents the expected output (1 for good coffee, 0 for bad coffee).
• dtype=np.float32 ensures numerical stability and compatibility with TensorFlow.

Step 3: Building the Model

model = tf.keras.Sequential([
    tf.keras.layers.Dense(3, activation='sigmoid', input_shape=(2,)),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

• Sequential() creates a linear stack of layers.
• Dense(3, activation='sigmoid', input_shape=(2,)) defines the hidden layer:
  • 3 neurons
  • Sigmoid activation function
  • Input shape of (2,) since we have two input features.
• Dense(1, activation='sigmoid') defines the output layer with 1 neuron and sigmoid activation.

Step 4: Training the Model

model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(X, y, epochs=500, verbose=0)

• compile() configures the model for training:
  • adam optimizer adapts the learning rate automatically.
  • binary_crossentropy is used for binary classification problems.
  • accuracy metric tracks how well the model classifies coffee samples.
• fit(X, y, epochs=500, verbose=0) trains the model for 500 epochs (iterations over data).

Step 5: Making Predictions

new_coffee = np.array([[205, 14]], dtype=np.float32)
prediction = model.predict(new_coffee)
print("Prediction (Probability of Good Coffee):", prediction)

• new_coffee contains a new sample (205°C, 14 min) to classify.
• model.predict(new_coffee) computes the probability of the coffee being good.
• The output is a probability (closer to 1 means good, closer to 0 means bad).

Forward Propagation Step-by-Step (NumPy Implementation)

We now implement forward propagation manually using NumPy to understand how TensorFlow executes it under the hood.

Initializing Weights and Biases

np.random.seed(42)  # For reproducibility
W1 = np.random.randn(2, 4)  # Weights for hidden layer (2 inputs -> 4 neurons)
b1 = np.random.randn(4)     # Bias for hidden layer
W2 = np.random.randn(4, 1)  # Weights for output layer (4 neurons -> 1 output)
b2 = np.random.randn(1)     # Bias for output layer

• np.random.randn() initializes weights and biases randomly from a normal distribution.
• W1 and b1 define the hidden layer parameters.
• W2 and b2 define the output layer parameters.

Forward Propagation Calculation

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

• This function applies the sigmoid activation function, which outputs values between 0 and 1.

def forward_propagation(X):
    Z1 = np.dot(X, W1) + b1  # Linear transformation (Hidden Layer)
    A1 = sigmoid(Z1)  # Activation function (Hidden Layer)
    Z2 = np.dot(A1, W2) + b2  # Linear transformation (Output Layer)
    A2 = sigmoid(Z2)  # Activation function (Output Layer)
    return A2

• np.dot(X, W1) + b1 computes the weighted sum of inputs for the hidden layer.
• sigmoid(Z1) applies the activation function to introduce non-linearity.
• np.dot(A1, W2) + b2 computes the weighted sum of outputs from the hidden layer.
• sigmoid(Z2) produces the final prediction.

# Testing with an example input
output = forward_propagation(np.array([[185, 10]]))
print(output)

This manually replicates TensorFlow's forward propagation but using pure NumPy.

Artificial General Intelligence (AGI)

AGI refers to AI that can perform any intellectual task a human can. Unlike current AI systems, AGI would adapt, learn, and generalize across different tasks without needing task-specific training.

Everyday Example: AGI vs. Narrow AI

• Narrow AI (Current AI): A chess-playing AI can defeat world champions but cannot drive a car.
• AGI: If a chess-playing AI was truly intelligent, it would learn how to drive just like a human without explicit programming.

Key Challenges in AGI

1. Transfer Learning: Current AI requires large amounts of data. Humans learn with few examples.
2. Common Sense Reasoning: AI struggles with simple logic like "If I drop a glass, it will break."
3. Self-Learning: AGI must improve without needing human intervention.

Is AGI Possible?

• Some scientists believe AGI is decades away, while others argue it may never happen.
• Brain-inspired architectures (like Neural Networks) might be a stepping stone toward AGI.

Neural Network Training and Activation Functions

• Understanding Loss Functions
  • Binary Crossentropy (BCE)
  • Mean Squared Error (MSE)
  • Categorical Crossentropy (CCE)
  • Sparse Categorical Crossentropy (SCCE)
  • Choosing the Right Loss Function
• Training Details Main Concepts
  • Epochs
  • Batch Size
  • Validation Data
• Activation Functions
  • 1. Why Do We Need Activation Functions?
  • 2. Common Activation Functions
    • Sigmoid (Logistic Function)
    • ReLU (Rectified Linear Unit)
    • Leaky ReLU
    • Softmax
    • Linear Activation
  • 3. Choosing the Right Activation Function
  • Softmax vs. Sigmoid: Key Differences
  • Improved Implementation of Softmax