Multilayer Networks, Gradient Descent, and Backpropagation

• 1. Multi-Layer Perceptrons (MLP)
  • 1.1 MLP Architecture and Parametric Notation
  • 1.2 Michael Nielsen’s MNIST Classifier Case Study
    • Parameter Count Breakdown:
• 2. Cost Function and Gradient Descent
  • 2.1 Desired Activations and the Mean Squared Error (MSE) Cost
    • Mean Squared Error (MSE) Cost Formulation:
  • 2.2 Mathematics of Gradient Descent and the Error Surface
    • Analytical Steepest Descent Derivation:
    • Parameter Update Equations:
  • 2.3 Computational Collapse of Brute-Force Finite Differences
    • Workload Quantification:
• 3. The Backpropagation Algorithm
  • 3.1 Analytical Derivation via the Chain Rule
  • 3.2 Local Gradient ($\delta$) Formulation
    • Backward Propagation of Errors to Hidden Layers:
  • 3.3 Complexity Comparison Matrix
• 4. Example Applications in Computer Vision
  • 4.1 Handwritten Digit Recognition (MNIST)
  • 4.2 Convolutional Neural Networks (LeNet / CNNs)
  • 4.3 Multi-Label Semantic Tagging (Clarifai)
• 5. When to Use Machine Learning?
  • 5.1 First Principles vs. Data-Driven ML
    • Drawbacks of Pure ML in Closed-Physics Regimes:
  • 5.2 Decision Matrix: First Principles vs. Machine Learning

This lecture note covers the second and most foundational phase of neural networks: the Multi-Layer Perceptron (MLP) architecture, the multi-dimensional geometry of loss functions, optimization via Gradient Descent, and the analytical derivation of the Backpropagation Algorithm via the calculus chain rule.

1. Multi-Layer Perceptrons (MLP)

While single perceptrons and single-layer linear classifiers can only resolve linearly separable tasks, Multi-Layer Perceptrons (MLPs) incorporating one or more Hidden Layers can approximate arbitrary continuous non-linear mappings between high-dimensional inputs and discrete or continuous outputs.

flowchart TD
    subgraph InputLayer["Input Layer (Layer 1)"]
        X1["x₁ (Pixel 1)"]
        X2["x₂ (Pixel 2)"]
        Xdots["..."]
        XN["x₇₈₄ (Pixel 784)"]
    end

    subgraph HiddenLayer["Hidden Layer (Layer 2)"]
        H1["Neuron 1 (σ)"]
        H2["Neuron 2 (σ)"]
        Hdots["..."]
        HM["Neuron 30 (σ)"]
    end

    subgraph OutputLayer["Output Layer (Layer 3/L)"]
        O0["Class 0"]
        O1["Class 1"]
        Odots["..."]
        O9["Class 9"]
    end

    InputLayer -->|"Weights W^(2), Biases b^(2)"| HiddenLayer
    HiddenLayer -->|"Weights W^(3), Biases b^(3)"| OutputLayer

    style InputLayer fill:#1a1a2e,stroke:#e94560,color:#fff
    style HiddenLayer fill:#16213e,stroke:#4cc9f0,color:#fff
    style OutputLayer fill:#0f3460,stroke:#e94560,color:#fff

1.1 MLP Architecture and Parametric Notation

A typical Multi-Layer Perceptron consists of three fundamental structural stages:

1. Input Layer (Layer 1): Accepts raw sensory input from the external environment. The nodes perform no mathematical activation; they simply broadcast feature values (e.g., $28 \times 28 = 784$ pixel intensities in an MNIST digit).
2. Hidden Layers (Layers $2 \dots L-1$): Intermediate layers containing activation units (such as Sigmoid or ReLU) that progressively extract hierarchically richer semantic representations (edges, textures, corners, parts).
3. Output Layer (Layer $L$): Produces the final network prediction. In an MNIST digit recognition task, this layer contains 10 output units representing class categories from $0$ to $9$.

1.2 Michael Nielsen’s MNIST Classifier Case Study

Michael Nielsen’s canonical handwritten digit classification network demonstrates this layered configuration:

• Input Layer: 784 neurons ($28 \times 28$ normalized pixel brightness vector).
• Hidden Layer: 30 neurons (fully connected).
• Output Layer: 10 neurons (one activation for each digit class $0$ to $9$).

Parameter Count Breakdown:

1. Weights:
   • Layer 1 to Layer 2: $784 \times 30 = 23,520$ weights
   • Layer 2 to Layer 3: $30 \times 10 = 300$ weights
   • Total Weights: $23,520 + 300 = 23,820$
2. Biases:
   • Hidden Layer: $30$ biases
   • Output Layer: $10$ biases
   • Total Biases: $30 + 10 = 40$
3. Total Trainable Parameters: $$\text{Total Parameters} = 23,820 + 40 = 23,860$$

2. Cost Function and Gradient Descent

A newly initialized network with random weights produces arbitrary, uncalibrated activations. Training the network entails adjusting parameters to minimize discrepancy between predicted outputs and ground-truth labels.

2.1 Desired Activations and the Mean Squared Error (MSE) Cost

For each training image $x$, ground-truth class labels are represented as one-hot encoded target vectors $\hat{\mathbf{a}}(x)$:

Prior to training, random initialization yields noisy output distributions far from ground truth:

Mean Squared Error (MSE) Cost Formulation:

For an individual training sample $x$, the quadratic cost $C_x$ measures the squared Euclidean distance between network activations $\mathbf{a}(x)$ and target vector $\hat{\mathbf{a}}(x)$:

$$C_x(\mathbf{w}, \mathbf{b}) = |\hat{\mathbf{a}}(x) - \mathbf{a}(x | \mathbf{w}, \mathbf{b})|^2 = \sum_{j} \left( \hat{a}_j(x) - a_j^L(x) \right)^2$$

Averaged across the full training dataset ($n = 60,000$ images):

$$C(\mathbf{w}, \mathbf{b}) = \frac{1}{n} \sum_{x} C_x(\mathbf{w}, \mathbf{b})$$

flowchart LR
    Init["1. Initialize Weights & Biases\nwith Random Values"] --> Forward["2. Forward Propagate Inputs\nCompute Network Activations a(x)"]
    Forward --> Cost["3. Evaluate Mean Squared Error\nCost Function C(w,b)"]
    Cost --> Opt["4. Gradient Descent Optimization\nUpdate Weights & Biases"]
    Opt --> Forward

    style Init fill:#1a1a2e,stroke:#e94560,color:#fff
    style Forward fill:#16213e,stroke:#4cc9f0,color:#fff
    style Cost fill:#0f3460,stroke:#e94560,color:#fff
    style Opt fill:#53354a,stroke:#e94560,color:#fff

2.2 Mathematics of Gradient Descent and the Error Surface

The objective is to locate the global or near-optimal local minimum on the 23,860-dimensional cost surface $C(\mathbf{w}, \mathbf{b})$.

Analytical Steepest Descent Derivation:

A differential parameter displacement $\Delta \mathbf{v} = [\Delta w_1, \dots, \Delta b_1, \dots]^T$ induces a first-order change in cost $\Delta C$:

$$\Delta C \approx \nabla C \cdot \Delta \mathbf{v}$$

Where $\nabla C$ is the gradient vector of partial derivatives:

$$\nabla C = \left[ \frac{\partial C}{\partial w_1}, \frac{\partial C}{\partial w_2}, \dots, \frac{\partial C}{\partial b_1}, \dots \right]^T$$

To enforce maximal decrease ($\Delta C < 0$), Cauchy-Schwarz inequality dictates selecting $\Delta \mathbf{v}$ antiparallel to $\nabla C$:

$$\Delta \mathbf{v} = -\eta \nabla C$$

Where $\eta > 0$ is the Learning Rate. Substituting this yields:

$$\Delta C \approx \nabla C \cdot (-\eta \nabla C) = -\eta |\nabla C|^2 \leq 0$$

Because $-\eta |\nabla C|^2$ is strictly non-positive, every gradient descent update step guarantees reducing or maintaining the objective loss!

Parameter Update Equations:

$$w_i \leftarrow w_i - \eta \frac{\partial C}{\partial w_i}$$ $$b_j \leftarrow b_j - \eta \frac{\partial C}{\partial b_j}$$

2.3 Computational Collapse of Brute-Force Finite Differences

To execute gradient descent, we must evaluate $23,860$ partial derivatives at every iteration.

Using the standard numerical Finite Differences approximation:

$$\frac{\partial C}{\partial w_k} \approx \frac{C(\mathbf{w} + \epsilon \mathbf{e}_k, \mathbf{b}) - C(\mathbf{w}, \mathbf{b})}{\epsilon}$$

Workload Quantification:

1. One image forward pass: $23,820$ multiplications.
2. Dataset evaluation ($60,000$ images) for one loss calculation $C(\mathbf{w}, \mathbf{b})$: $$60,000 \times 23,820 \approx 1.43 \times 10^9 \text{ multiplications}$$
3. Perturbing $23,860$ parameters individually requires evaluating the dataset 23,861 times: $$\text{Workload for 1 Gradient Step} = 23,861 \times (1.43 \times 10^9) \approx \mathbf{3.4 \times 10^{13}} \text{ multiplications!}$$

Critical Bottleneck: On supercomputers executing billions of operations per second, a single step would take days. Brute-force numerical differentiation is completely intractable for deep learning.

3. The Backpropagation Algorithm

The breakthrough that unlocked scalable neural network training is the Backpropagation Algorithm, which slashes gradient computation time by a factor of $10,000$.

3.1 Analytical Derivation via the Chain Rule

Backpropagation computes exact analytical derivatives in a single backward sweep using calculus chain rule. Let us derive the partial derivative for an output layer weight $w_{11}^{(4)}$:

Applying the chain rule:

$$\frac{\partial C_x}{\partial w_{ji}^L} = \frac{\partial C_x}{\partial a_j^L} \cdot \frac{\partial a_j^L}{\partial z_j^L} \cdot \frac{\partial z_j^L}{\partial w_{ji}^L}$$

Evaluating each factor analytically:

1. Loss with respect to Activation: $$C_x = \sum_k (a_k^L - \hat{a}_k)^2 \implies \frac{\partial C_x}{\partial a_j^L} = 2(a_j^L - \hat{a}_j)$$
2. Activation with respect to Net Input (Sigmoid Derivative): $$a_j^L = \sigma(z_j^L) \implies \frac{\partial a_j^L}{\partial z_j^L} = \sigma’(z_j^L) = \sigma(z_j^L)(1 - \sigma(z_j^L)) = a_j^L (1 - a_j^L)$$
3. Net Input with respect to Weight: $$z_j^L = \sum_k w_{jk}^L a_k^{L-1} + b_j^L \implies \frac{\partial z_j^L}{\partial w_{ji}^L} = a_i^{L-1}$$

Combining these terms:

$$\frac{\partial C_x}{\partial w_{ji}^L} = \underbrace{\left[ 2(a_j^L - \hat{a}j) \cdot a_j^L (1 - a_j^L) \right]}{\text{Local Gradient } \delta_j^L} \cdot a_i^{L-1}$$

3.2 Local Gradient ($\delta$) Formulation

The bracketed term defines the Local Gradient ($\delta_j^L$) of unit $j$ in layer $L$:

$$\delta_j^L = \frac{\partial C_x}{\partial z_j^L} = 2(a_j^L - \hat{a}_j) \cdot a_j^L (1 - a_j^L)$$

This reduces all parameter derivatives into modular two-factor products:

$$\frac{\partial C_x}{\partial w_{jk}^{(l)}} = \delta_j^{(l)} a_k^{(l-1)}$$ $$\frac{\partial C_x}{\partial b_j^{(l)}} = \delta_j^{(l)}$$

Backward Propagation of Errors to Hidden Layers:

Given downstream errors $\delta^{l+1}$, the error $\delta^l$ at hidden layer $l$ propagates backward through the transpose weight matrix:

$$\delta_j^l = \left( \sum_k \delta_k^{l+1} w_{kj}^{l+1} \right) a_j^l (1 - a_j^l)$$

flowchart RL
    subgraph BackpropFlow["Error & Gradient Flow (Backward)"]
        DL["Output Errors: δ^(L)"] -->|"Matrix Product (W^(L))^T"| DL1["Hidden Errors: δ^(L-1)"]
        DL1 -->|"Matrix Product (W^(L-1))^T"| DL2["Prior Hidden Errors: δ^(2)"]
    end

    style DL fill:#e94560,stroke:#fff,color:#fff
    style DL1 fill:#0f3460,stroke:#4cc9f0,color:#fff
    style DL2 fill:#16213e,stroke:#4cc9f0,color:#fff

3.3 Complexity Comparison Matrix

Summary: Backpropagation slashes gradient complexity by 4 orders of magnitude ($10^4$), turning impossible multi-day computations into sub-second matrix passes.

4. Example Applications in Computer Vision

4.1 Handwritten Digit Recognition (MNIST)

Trained MLPs correctly classify heavily stylized and noisy digits with high confidence:

4.2 Convolutional Neural Networks (LeNet / CNNs)

In early computer vision, spatial feature extractors (e.g., Sobel or Gaussian filters) were handcrafted. In Yann LeCun’s (1998) Convolutional Neural Networks:

1. Kernel filter coefficients ($k_1 \dots k_5$) serve as trainable network weights.
2. Backpropagation automatically learns optimal spatial visual filters tailored to the visual task.

4.3 Multi-Label Semantic Tagging (Clarifai)

Modern deep networks extract rich multi-concept descriptors from unconstrained visual scenes:

5. When to Use Machine Learning?

While deep neural networks are remarkably versatile, applying machine learning to phenomena governed by well-known physical laws introduces unnecessary computational overhead and opacity.

5.1 First Principles vs. Data-Driven ML

Consider calculating the displacement $s$ of a falling body over time $t$:

• First Principles (Newtonian Physics): $$s = ut + \frac{1}{2}at^2$$ Requires zero training data, evaluates instantly, and delivers foundational physical insight ($a = \text{gravity}$).
• Data-Driven ML: Dropping hundreds of balls, logging noisy time records, and fitting thousands of parameters with stochastic gradient descent.

Drawbacks of Pure ML in Closed-Physics Regimes:

1. Computational & Resource Inefficiency: Massive data collection and GPU cycles are wasted relearning a known exact formula.
2. Zero Scientific Insight (Black-Box): The network models input-output correlation without conveying physical understanding of gravitational mechanics.
3. The Last Mile Problem: Pure data models reach $90-95%$ accuracy quickly, but attaining the $99.99%$ reliability required by safety-critical vision systems demands increasingly prohibitive data collection.

5.2 Decision Matrix: First Principles vs. Machine Learning

Golden Engineering Rule (Symbiotic Coexistence): The most robust computer vision systems leverage first principles as far as analytical modeling permits, and transition to machine learning precisely where hand-crafted physical rules reach their descriptive limit.