Neural Network Architecture and Forward Propagation

Structure of Neural Networks

Neural networks are computational models inspired by the human brain. They consist of layers of interconnected nodes (neurons) that process information. A typical neural network has three types of layers:

• Input Layer: Receives the initial data
• Hidden Layers: Process the information through weighted connections
• Output Layer: Produces the final result

A neural network consists of layers of interconnected neurons

Each connection between neurons has a weight, and each neuron applies an activation function to its weighted sum of inputs. The forward propagation process is how data flows through the network to produce predictions.

Mathematical Representation

Mathematically, each neuron in a neural network computes a weighted sum of its inputs, adds a bias term, and then applies an activation function:

z = w₁x₁ + w₂x₂ + ... + wₙxₙ + b
a = σ(z)

Where:

• x₁, x₂, ..., xₙ are the inputs to the neuron
• w₁, w₂, ..., wₙ are the weights associated with each input
• b is the bias term
• z is the weighted sum (pre-activation)
• σ is the activation function
• a is the output of the neuron (post-activation)

For a layer of neurons, we can represent this computation using matrix operations:

Z = XW + b
A = σ(Z)

Where:

• X is the input matrix (or output from the previous layer)
• W is the weight matrix for the current layer
• b is the bias vector
• Z is the pre-activation matrix
• A is the post-activation matrix (output of the layer)

Activation Functions

Activation functions introduce non-linearity into the network. Without them, a neural network would just be a series of linear transformations, which could be collapsed into a single linear transformation.

Common activation functions and their derivatives

Common activation functions include:

• Sigmoid: σ(z) = 1/(1+e^(-z))
  • Squashes values between 0 and 1
  • Derivative: σ'(z) = σ(z)(1-σ(z))
• ReLU (Rectified Linear Unit): f(z) = max(0, z)
  • Returns 0 for negative inputs, z for positive inputs
  • Derivative: f'(z) = 0 if z < 0, 1 if z > 0
• Tanh: tanh(z) = (e^z - e^(-z))/(e^z + e^(-z))
  • Squashes values between -1 and 1
  • Derivative: tanh'(z) = 1 - tanh²(z)

The choice of activation function can significantly impact the performance of a neural network. ReLU is commonly used in hidden layers due to its computational efficiency and ability to mitigate the vanishing gradient problem.

Forward Propagation

Forward propagation is the process of passing input data through the network to generate predictions. It involves the following steps:

1. Input values are fed into the network
2. Each neuron computes a weighted sum of its inputs
3. An activation function is applied to introduce non-linearity
4. The output becomes input for the next layer
5. The process continues until the output layer

For a neural network with L layers, forward propagation can be represented as:

A⁰ = X  # Input layer
For l = 1 to L:
    Z^l = A^(l-1)W^l + b^l
    A^l = σ^l(Z^l)
Y_pred = A^L  # Output layer

Where:

• A⁰ is the input data
• Z^l is the pre-activation for layer l
• A^l is the post-activation for layer l
• W^l is the weight matrix for layer l
• b^l is the bias vector for layer l
• σ^l is the activation function for layer l
• Y_pred is the final output (prediction)